wise-white•2y ago
How can i delay the Cheerio Crawler requests?
I have a cheerio crawler that works and basically "gets the sitemap" of a website. My issue is that due to the fact that the enqueueLinks() uses requests in such a fast manner that I get blocked from several sites when using the crawler as they believe that i am trying to spam requests/attack them (basically it leads to me having my ip banned).
Can i make it that the requests between links are delayed a bit so it doesn't seem like i am an attacker because the number of requests that the crawler makes to a website is really high in a short manner of time?
1 Reply