Crawl Speed

Lets say i have one session that makes 15 requests. I do not like that it finishes crawl in 25 seconds. How to stretch crawl duration / slow down speed of crawl for small ones? max_requests_per_minute is useless in this situation right? What options i have?

Mantisus•10/28/25, 8:02 AM

max_requests_per_minutemax_requests_per_minute is exactly the parameter you need.

DrogbataOP•10/28/25, 8:03 AM

but isnt it rolling 60 seconds window? what if crawl duration is less then 60

Mantisus•10/28/25, 8:06 AM

With this setting, you don't run all requests at once and wait a minute.

Crawlee will try to keep even time intervals between requests.

Mantisus•10/28/25, 8:07 AM

20 requests with max_requests_per_minute=10max_requests_per_minute=10 will take about 2 minutes to complete, with a 6-second interval between requests.

DrogbataOP•10/28/25, 8:16 AM

oh i get it, thank you