ambitious-aqua•2y ago

Gracefully closing the crawler with keepalive flag true

I'm using puppeteer crawler with keepAlive as true and crawler.run() (without await). This runs the crawler infinitely and if I insert new requests to the requests queue, they get processed. (I'm using non persisted request queue) What I want is to gracefully close the crawler, as in If I get a signal to close, I want to process all the pending requests in the requests queue first and then kill the crawler. Right now If I do crawler.teardown(), it abruptly closes the crawler instances without processing the pending requests.

2 Replies

Pepa J•2y ago

Hi @makditeam : Maybe you want to define isFinishedFunction instead? https://docs.apify.com/sdk/js/docs/2.3/typedefs/autoscaled-pool-options#isfinishedfunction

// ...
    autoscaledPoolOptions: {
        isFinishedFunction: () => {
           return wasAbortedAndAllRequestsHandled()
        },
    },
// ...

// ...
    autoscaledPoolOptions: {
        isFinishedFunction: () => {
           return wasAbortedAndAllRequestsHandled()
        },
    },
// ...

AutoscaledPoolOptions | SDK for JavaScript | Apify Documentation

Properties

Lukas Krivka•2y ago

You can check there if the queue is empty

Gracefully closing the crawler with keepalive flag true

Did you find this page helpful?