quickest-silver•2y ago

How to catch/access the errors in the handler function for failed request

So in many cases, the request fails with 429 status code or sometimes because of ERR_TIMED_OUT error etc. (Example: Reclaiming failed request back to the list or queue. Request blocked - received 429 status code.) How do I catch these errors in the handler function so that I can take some kind of action on it? Any help would be appreciated! 🙂

2 Replies

Lukas Krivka•2y ago

That depends on what error it is actually. Crawlee by default auto-retries on certain statuses, you can disable that via

sessionPoolOptions: {
    blockedStatusCodes: []
}

sessionPoolOptions: {
    blockedStatusCodes: []
}

Also, you can react to any error in the errorHandler but keep in mind that the function passed there must not throw

quickest-silverOP•2y ago

Thanks @Lukas Krivka 🙂 I ended up using postNavigationHooks to handle error codes like 429 and write a backoff logic for rate limits

How to catch/access the errors in the handler function for failed request

Did you find this page helpful?