Vverbal-lime
Hey everyone, would love it if we had a
Hey everyone, would love it if we had a way to shutdown crawlers from inside request handler, I went through the docs today and the only way to do it right now is via the
I have found a workaround for this by setting a
crawlercrawler itself either using crawler.teardown()crawler.teardown() or crawler.requestQueue.drop()crawler.requestQueue.drop() (Not sure about this one), the main use case for it being saving on proxy costs/stopping crawlers from redundantly scraping data or even some other arbritrary conditions. I have found a workaround for this by setting a
shutdownshutdown flag in a state or even a variable and checking for it inside the handlers and if its true, just doing a return;return;(to empty out the queue) while this works it does add in a lot of noise in logs (also in the code) because we need to log that we are skipping them because of this flag for debugging purposes and I wish it would be handled a little more gracefully in the scraper instead of every request handler checking for it
