Nnursing-lime
Override default terminal logging
I have a trivial nuance if anyone could point me in the right direction. I have a crawler setup using the CheerioCrawler where I am pulling urls from a database in batches and adding them to the requestQueue.
Everything is working fine however for my specific application I do not want to retry any urls if any type of error occurs (ssl, status codes, etc) mainly due to proxy rotation etc the details are not important.
Which brings me to my problem. Is there any way to override crawlee's default terminal logging as it is clogging up the terminal with errors and stack traces due to reaching maximum retries.
ERROR CheerioCrawler: Request failed and reached maximum retries. RequestError: The proxy server rejected the request with status code 502 (Bad Gateway):
ERROR CheerioCrawler: Request failed and reached maximum retries. RequestError: 18530000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:c:\ws\deps\openssl\openssl\ssl\record\ssl3_record.c:355:
with long stack traces
when creating the crawler I have the maxRequestRetries set to 0
I have also set the noRetry option to true in each requests constructor
however I am still getting the maximum retries ERROR logged to terminal
Even if I could just turn off the stack traces.
Thanks
Everything is working fine however for my specific application I do not want to retry any urls if any type of error occurs (ssl, status codes, etc) mainly due to proxy rotation etc the details are not important.
Which brings me to my problem. Is there any way to override crawlee's default terminal logging as it is clogging up the terminal with errors and stack traces due to reaching maximum retries.
ERROR CheerioCrawler: Request failed and reached maximum retries. RequestError: The proxy server rejected the request with status code 502 (Bad Gateway):
ERROR CheerioCrawler: Request failed and reached maximum retries. RequestError: 18530000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:c:\ws\deps\openssl\openssl\ssl\record\ssl3_record.c:355:
with long stack traces
when creating the crawler I have the maxRequestRetries set to 0
I have also set the noRetry option to true in each requests constructor
however I am still getting the maximum retries ERROR logged to terminal
Even if I could just turn off the stack traces.
Thanks
This is the official developer community of Apify and Crawlee.
13,923Members
Resources
Similar Threads
Was this page helpful?
