Any suggestions for improving the speed of the crawling run?

Hello there!

Beside reducing the scope of what is being crawled, for example of number of pages, etc, what can we do in order to accelerate the run?

Any suggestions are welcomed, I'm simply curious.

Hall•9/22/24, 10:15 PM

View post on community site

This post has been pushed to the community knowledgebase. Any replies in this thread will be synced to the community site.

Apify Community

lemurio•9/27/24, 8:28 AM

hey, aside from obvious improvements like allocating more memory or increasing maximum concurrency, if you're using a browser for scraping, try to find ways to replace parts of the process with simple HTTP calls, without relying on the browser navigation. You can scrape the website's internal or public JSON API or the site's pure HTML this way. Use the browser only when necessary.

military-pink•9/27/24, 5:47 PM

Basically increasing parallelism and concurrency is the first factor.
But in some cases its worth it to optimize request and parsing logic.

military-pink•9/27/24, 5:47 PM

@gablabelle