Apify & CrawleeA&CApify & Crawlee
Powered by
dangerous-fuchsiaD
Apify & Crawlee•3y ago•
7 replies
dangerous-fuchsia

CheerioCrawler Timeout after 320 Seconds Error/Exception

In some of our CheerioCrawler actors, we continue to get some random timeout errors after 320 seconds that cause them to crash. This is an example of the error:

2023-06-08T07:28:54.464Z ERROR CheerioCrawler: An exception occurred during handling of failed request. This places the crawler and its underlying storages into an unknown state and crawling will be terminated. This may have happened due to an internal error of Apify's API or due to a misconfigured crawler.
2023-06-08T07:28:54.464Z ERROR CheerioCrawler: An exception occurred during handling of failed request. This places the crawler and its underlying storages into an unknown state and crawling will be terminated. This may have happened due to an internal error of Apify's API or due to a misconfigured crawler.


This is an example of another error message that occurs:
2023-06-08T07:28:54.467Z   Error: Handling request failure of http://www.yellowpages.com/search?search_terms=9724388344 (VrMLvREYl5fRBJQ) timed out after 320 seconds.
2023-06-08T07:28:54.469Z       at Timeout._onTimeout (/usr/src/app/node_modules/@apify/timeout/index.js:62:68)
2023-06-08T07:28:54.471Z       at listOnTimeout (node:internal/timers:559:17)
2023-06-08T07:28:54.473Z       at processTimers (node:internal/timers:502:7)
2023-06-08T07:28:54.475Z ERROR CheerioCrawler:AutoscaledPool: runTaskFunction failed.
2023-06-08T07:28:54.478Z   Error: Handling request failure of http://www.yellowpages.com/search?search_terms=9724388344 (VrMLvREYl5fRBJQ) timed out after 320 seconds.
2023-06-08T07:28:54.480Z       at Timeout._onTimeout (/usr/src/app/node_modules/@apify/timeout/index.js:62:68)
2023-06-08T07:28:54.482Z       at listOnTimeout (node:internal/timers:559:17)
2023-06-08T07:28:54.484Z       at processTimers (node:internal/timers:502:7)
2023-06-08T07:28:54.467Z   Error: Handling request failure of http://www.yellowpages.com/search?search_terms=9724388344 (VrMLvREYl5fRBJQ) timed out after 320 seconds.
2023-06-08T07:28:54.469Z       at Timeout._onTimeout (/usr/src/app/node_modules/@apify/timeout/index.js:62:68)
2023-06-08T07:28:54.471Z       at listOnTimeout (node:internal/timers:559:17)
2023-06-08T07:28:54.473Z       at processTimers (node:internal/timers:502:7)
2023-06-08T07:28:54.475Z ERROR CheerioCrawler:AutoscaledPool: runTaskFunction failed.
2023-06-08T07:28:54.478Z   Error: Handling request failure of http://www.yellowpages.com/search?search_terms=9724388344 (VrMLvREYl5fRBJQ) timed out after 320 seconds.
2023-06-08T07:28:54.480Z       at Timeout._onTimeout (/usr/src/app/node_modules/@apify/timeout/index.js:62:68)
2023-06-08T07:28:54.482Z       at listOnTimeout (node:internal/timers:559:17)
2023-06-08T07:28:54.484Z       at processTimers (node:internal/timers:502:7)

I have tried wrapping the code in the failedRequestHandler in a try/catch block, but that doesn't provide any additional information.

I manually resurrected two recent jobs that failed, and their ids are
3NgQTkqcodOxjuDGZ
3NgQTkqcodOxjuDGZ
and
p185ASvE8SiStkvZX
p185ASvE8SiStkvZX
.

Any insight would be greatly appeciated, as this is impacting production. Thank you!
Apify & Crawlee banner
Apify & CrawleeJoin
This is the official developer community of Apify and Crawlee.
13,739Members
Resources
Was this page helpful?

Similar Threads

Recent Announcements
Recent Announcements
ellativity

**The Apify $1M Challenge is over!** For everyone who joined yesterday’s Award Ceremony livestream for the Apify $1M Challenge, thank you for your enthusiastic drumrolls in the chat and positive vibes. We were really feeling the excitement and celebratory mood! If you missed the stream or just want to rewatch the key moments again, here’s the replay link https://www.youtube.com/watch?v=eEDV-5X43Gg (ngl, the replay is not the same without your live chat) And, if you didn’t check the email that should have landed in your inboxes, we’d love to hear about your experience of participating in the Apify $1M Challenge. **<a:alerthulk:1468892073917939713> Win one of five $100 Visa gift cards by completing the end-of-challenge survey here: https://apify.typeform.com/to/mjoMaZqD** Thank you again to everyone who participated in any capacity. The past 3 months have been a wild ride and we feel so grateful to have been on this adventure with y’all. We mean every word when we say how much you’ve impressed us. Thank you all from the bottom of our hearts. <a:keanuthanks:1430839059655426068> Saurav and Ella xoxo PS - if you just want to jump to the spoilers, a full list of winners is available at https://apify.com/challenge 🏆

ellativity · 5d ago

ellativity

**You are invited** ... to celebrate all the achievements of the Apify $1M Challenge with us on Wednesday, February 4 at **8 AM PT / 11 AM ET / 4 PM GMT / 5 PM CET / 9:30 PM IST / 12 AM +1d CST** We will be announcing winners of the Grand Prizes, as well as regional winners and much more, with especially good news for all participating developers. 🏆 We look forward to sharing with you all! 🎉 More info here: https://luma.com/6c1493t0

ellativity · 2w ago

ellativity

Hi @everyone 👋 I'm hanging out in https://discord.com/channels/801163717915574323/1430491198145167371 for the next 45 min, if you want to discuss the end of the challenge or anything else.

ellativity · 2w ago

Similar Threads

CheerioCrawler headerGenerator help
NeoNomadeNNeoNomade / crawlee-js
13mo ago
change INFO CheerioCrawler
wispy-oliveWwispy-olive / crawlee-js
3y ago
How can i change request timeout to 10 seconds instead of 30 seconds
ill-bronzeIill-bronze / crawlee-js
3y ago
Node-cron with CheerioCrawler
popular-magentaPpopular-magenta / crawlee-js
3y ago