Crawl Speed
Lets say i have one session that makes 15 requests. I do not like that it finishes crawl in 25 seconds. How to stretch crawl duration / slow down speed of crawl for small ones? max_requests_per_minute is useless in this situation right? What options i have?
4 Replies
max_requests_per_minute is exactly the parameter you need.but isnt it rolling 60 seconds window? what if crawl duration is less then 60
With this setting, you don't run all requests at once and wait a minute.
Crawlee will try to keep even time intervals between requests.
20 requests with
max_requests_per_minute=10 will take about 2 minutes to complete, with a 6-second interval between requests.oh i get it, thank you