yappiest-sapphire•2y ago

crawler process not exiting after teardown is called.

I am running a Playwright crawler and I call await crawler.teardown() after some condition is met, Issue with this approach is that the crawlee process hangs indefinitely and doesn't exit even after a while. I have to force stop it using CTRL + C, on searching github, it looks like its an issue with some timeouts but I havent set one anywhere. here's how I am initializing my crawler:

export const crawler = new PlaywrightCrawler({
    requestHandler:router,
    maxRequestsPerMinute:10,
})

export const crawler = new PlaywrightCrawler({
    requestHandler:router,
    maxRequestsPerMinute:10,
})

and I have a playwright config where I have set the default timeout

import { defineConfig } from '@playwright/test';

export default defineConfig({
    timeout: 3000,
});

import { defineConfig } from '@playwright/test';

export default defineConfig({
    timeout: 3000,
});

3 Replies

yappiest-sapphireOP•2y ago

it just logs this at the end and then hangs indefinitely

2023-12-06T03:29:29.063Z info: Finished! Total 7 requests: 7 succeeded, 0 failed. {
  "terminal": true
}

2023-12-06T03:29:29.063Z info: Finished! Total 7 requests: 7 succeeded, 0 failed. {
  "terminal": true
}

MEE6•2y ago

@AltairSama2 just advanced to level 5! Thanks for your contributions! 🎉

yappiest-sapphireOP•2y ago

I fixed it by removing the return statement in my route handler, I guess it was causing issues with teardown's promise, I'm not sure why though

if (state.duplicate_count <= 0 || state.page_count > state.num_pages) {
            log.debug(
                `10 duplicate entries encountered in a row, Shutting down....`
            );
            await crawler.teardown();
            return; //removing this
        }

if (state.duplicate_count <= 0 || state.page_count > state.num_pages) {
            log.debug(
                `10 duplicate entries encountered in a row, Shutting down....`
            );
            await crawler.teardown();
            return; //removing this
        }

crawler process not exiting after teardown is called.

Did you find this page helpful?