4 replies

Hey everyone, would love it if we had a

Hey everyone, would love it if we had a way to shutdown crawlers from inside request handler, I went through the docs today and the only way to do it right now is via the

crawler

crawler

itself either using

crawler.teardown()

crawler.teardown()

crawler.requestQueue.drop()

crawler.requestQueue.drop()

(Not sure about this one), the main use case for it being saving on proxy costs/stopping crawlers from redundantly scraping data or even some other arbritrary conditions.

I have found a workaround for this by setting a

shutdown

shutdown

flag in a state or even a variable and checking for it inside the handlers and if its true, just doing a

return;

return;

(to empty out the queue) while this works it does add in a lot of noise in logs (also in the code) because we need to log that we are skipping them because of this flag for debugging purposes and I wish it would be handled a little more gracefully in the scraper instead of every request handler checking for it

Apify & Crawlee•3y ago•

4 replies

verbal-lime

Hey everyone, would love it if we had a

Hey everyone, would love it if we had a way to shutdown crawlers from inside request handler, I went through the docs today and the only way to do it right now is via the

crawler

crawler

itself either using

crawler.teardown()

crawler.teardown()

crawler.requestQueue.drop()

crawler.requestQueue.drop()

shutdown

shutdown

flag in a state or even a variable and checking for it inside the handlers and if its true, just doing a

return;

return;

Hey everyone, would love it if we had a

Similar Threads

Hey everyone, would love it if we had a

Similar Threads

Similar Threads

Similar Threads