Max requests per second
Hello!
I would like to know, is there, like for the maxRequestsPerMinute / maxTasksPerMinute an option but for second?
If not, what would be the easiest way to implement this? Always waiting 1s in the request handler and relying on maxConcurrency?
Thanks!
2 Replies
@MrSquaare just advanced to level 1! Thanks for your contributions! 🎉
hey, there is currently no option for second, you could consider setting maxConcurrency to some reasonable number.
this article could help: https://crawlee.dev/js/docs/guides/scaling-crawlers
Scaling our crawlers | Crawlee for JavaScript · Build reliable cra...
Crawlee helps you build and maintain your crawlers. It's open source, but built by developers who scrape millions of pages every day for a living.