conventional-tan•2y ago

Per request timeout

I can set an overall timeout for a crawler with requestHandlerTimeoutSecs like so:

const crawler = new PlaywrightCrawler({
    requestHandler: router,
    browserPoolOptions: {
        maxOpenPagesPerBrowser: 4,
    },
    requestHandlerTimeoutSecs: 3600,
})

const crawler = new PlaywrightCrawler({
    requestHandler: router,
    browserPoolOptions: {
        maxOpenPagesPerBrowser: 4,
    },
    requestHandlerTimeoutSecs: 3600,
})

Is there a way to set a timeout per page, rather than an overall timeout? E.g. if there is an exception being caught and the crawl then hangs, I was to timeout after 30 seconds on that page, but have a much longer timeout for the run overall

1 Reply

dependent-tan•2y ago

requestHandlerTimeoutSecs applies to each page, not the overall crawl. It's the timeout for the requestHandler function. https://crawlee.dev/api/3.5/basic-crawler/interface/BasicCrawlerOptions#requestHandlerTimeoutSecs

BasicCrawlerOptions | API | Crawlee

Per request timeout

Did you find this page helpful?