Apify
Crawlee
GitHub
Apify Discord Mirror
View all posts
Related posts
Was this helpful?
๐
๐
๐
Powered by
Hall
Inactive
Updated 5 months ago
0
Follow
per-site interval between requests?
per-site interval between requests?
Inactive
0
Follow
At a glance
n
new_in_town
2 years ago
ยท
Imagine the request queue of Crawlee (PlaywrightCrawler) containing URLs of two (or more) sites:
example.com/url1
another-site.com/url2
example.com/url3
another-site.com/url4
...
I would like to configure Crawlee to have per-site interval between requests. For the above example it means:
example.com: 20 sec (or more) between requests
another-site.com: 60 sec (or more) between requests
How to do this with Crawlee?
H
1 comment
Share
Open in Discord
H
HonzaS
2 years ago
just add put wait function in
preNavigationHooks
Add a reply
Sign up and join the conversation on Discord
Join on Discord