sensitive-blue•2y ago
IF a request times out, continue crawling
How to handle this scenario ? I have 100 requests, 2 fail after retrying but I want the crawler to not throw an error, thus failing. I want it to continue crawling the rest of the URL's.
Is it, with the help of failedRequestHandler ?
Thanks a lot !
3 Replies
Hey @Ayar, feel free to ask Crawlee Helper: https://chat.openai.com/g/g-a5WgfrNFC-crawlee-helper
If you still don't get the answer would love to help.
ChatGPT
ChatGPT - Crawlee Helper
Expert in Crawlee web scraping library, provides detailed answers from documentation.
Hi, Can you please provide some reproduction / link to your run on the platform.
By default Actor should continue crawling the rest of the URL's.
extended-salmon•2y ago
Yes, the request handler will retry the requests ( 3 times I think default ). It can seem like it fails but it's just a warning. If the failed urls are back to back it can take a long time depending on the timeout you have set for the page.goto()