Apify Discord Mirror

Updated 11 months ago

Is there a gentle way to run a crawler for x minutes?

At a glance

The community member is looking for a way to stop crawling after a certain number of requests using the maxRequestsPerCrawl option. Another community member suggests there may be a "gentle" way to run a crawler for a specific duration of time. A third community member provides a link to Apify's documentation, which mentions the ability to set a timeout, but the original poster clarifies they are not using Apify and are instead using crawlee.dev locally. There is no explicitly marked answer in the comments.

Useful resources
I understand there's a way to stop crawling after x number of requests with maxRequestsPerCrawl
S
O
3 comments
Is there a gentle way to run a crawler for x minutes?
Thanks Oleg but I'm not using Apify, I'm running crawlee.dev locally.
Add a reply
Sign up and join the conversation on Discord