How to cleanly cancel/abort a queued request before network call & handler without triggering error
🍩CheerioCrawler
Hey everyone!
I'm using
The goal is to completely discard the request so it:
- Does not make the network request.
- Does not execute the
- Does not count as a failed request or log an error.
1. Throwing a custom error in
I also swallow the error in
Problem: Crawlee still marks the request as FAILED, logs
2. Using
Problem: This skips the HTTP request, but
Is there an official/idiomatic way to completely cancel an enqueued request so it is silently discarded without:
- Making the network request.
- Running the route/request handler.
- Being treated as a failed request.
Thanks!
I'm using
CheerioCrawlerCheerioCrawler and need a way to conditionally drop/abort requests that are already in the RequestQueueRequestQueue when a runtime condition is met.The goal is to completely discard the request so it:
- Does not make the network request.
- Does not execute the
requestHandlerrequestHandler.- Does not count as a failed request or log an error.
What I tried
1. Throwing a custom error in preNavigationHookspreNavigationHooks
I also swallow the error in
errorHandlererrorHandler and failedRequestHandlerfailedRequestHandler.Problem: Crawlee still marks the request as FAILED, logs
Request failed and reached maximum retriesRequest failed and reached maximum retries, and increments failed request metrics.2. Using request.skipNavigation = truerequest.skipNavigation = true
Problem: This skips the HTTP request, but
requestHandlerrequestHandler still executes unless I add manual guard logic.Question
Is there an official/idiomatic way to completely cancel an enqueued request so it is silently discarded without:
- Making the network request.
- Running the route/request handler.
- Being treated as a failed request.
Thanks!
