Per request timeout

I can set an overall timeout for a crawler with requestHandlerTimeoutSecsrequestHandlerTimeoutSecs like so:

const crawler = new PlaywrightCrawler({
    requestHandler: router,
    browserPoolOptions: {
        maxOpenPagesPerBrowser: 4,
    },
    requestHandlerTimeoutSecs: 3600,
})

const crawler = new PlaywrightCrawler({
    requestHandler: router,
    browserPoolOptions: {
        maxOpenPagesPerBrowser: 4,
    },
    requestHandlerTimeoutSecs: 3600,
})

Is there a way to set a timeout per page, rather than an overall timeout? E.g. if there is an exception being caught and the crawl then hangs, I was to timeout after 30 seconds on that page, but have a much longer timeout for the run overall

Apify & Crawlee•3y ago•

1 reply

exclusive-coral

Per request timeout

I can set an overall timeout for a crawler with requestHandlerTimeoutSecsrequestHandlerTimeoutSecs like so:

const crawler = new PlaywrightCrawler({
    requestHandler: router,
    browserPoolOptions: {
        maxOpenPagesPerBrowser: 4,
    },
    requestHandlerTimeoutSecs: 3600,
})

const crawler = new PlaywrightCrawler({
    requestHandler: router,
    browserPoolOptions: {
        maxOpenPagesPerBrowser: 4,
    },
    requestHandlerTimeoutSecs: 3600,
})

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,851 Members

Join

Per request timeout

Per request timeout

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads