Sscattered-teal
Gracefully closing the crawler with keepalive flag true
I'm using puppeteer crawler with
This runs the crawler infinitely and if I insert new requests to the requests queue, they get processed.
(I'm using non persisted request queue)
What I want is to gracefully close the crawler, as in If I get a signal to close, I want to process all the pending requests in the requests queue first and then kill the crawler.
Right now If I do
keepAlivekeepAlive as truetrue and crawler.run()crawler.run() (without await).This runs the crawler infinitely and if I insert new requests to the requests queue, they get processed.
(I'm using non persisted request queue)
What I want is to gracefully close the crawler, as in If I get a signal to close, I want to process all the pending requests in the requests queue first and then kill the crawler.
Right now If I do
crawler.teardown()crawler.teardown(), it abruptly closes the crawler instances without processing the pending requests.
