correct-apricot•17mo ago
Retry after 30 seconds for failed requests
Is there a way that we can retry the failed requests after 30 seconds for cheerioCrawler
4 Replies
@Sunny just advanced to level 1! Thanks for your contributions! 🎉
You have to set up requestHandlerTimeoutSecs: 30,
correct-apricotOP•17mo ago
@Saurav Jain But the definition says - Timeout in which the function passed as requestHandler needs to finish, in seconds.
it is the total timeout for the requestHandler right ?
@Saurav Jain Thanks i think we need to use errorHandler function
errorHandler: async ({request}) => {
log.info(
Error occurred for URL - ${request.url}: ${request.errorMessages});
log.info(Waiting 30 seconds for URL - ${request.url});
// Wait for 30 seconds before retrying
await new Promise(resolve => setTimeout(resolve, 30000));
// Return true to indicate that the error should be retried
return true;
}@Sunny nice work!