Apify Discord Mirror

Updated 5 months ago

Per request timeout

At a glance

The community member is using the PlaywrightCrawler and wants to set a timeout per page, rather than an overall timeout. The community member is currently using requestHandlerTimeoutSecs to set an overall timeout of 3600 seconds.

In the comments, another community member explains that requestHandlerTimeoutSecs applies to each page, not the overall crawl. It is the timeout for the requestHandler function.

Useful resources

ccdslash

I can set an overall timeout for a crawler with requestHandlerTimeoutSecs like so:

Plain Text

const crawler = new PlaywrightCrawler({
    requestHandler: router,
    browserPoolOptions: {
        maxOpenPagesPerBrowser: 4,
    },
    requestHandlerTimeoutSecs: 3600,
})

Is there a way to set a timeout per page, rather than an overall timeout? E.g. if there is an exception being caught and the crawl then hangs, I was to timeout after 30 seconds on that page, but have a much longer timeout for the run overall

1 comment

HHamza

requestHandlerTimeoutSecs applies to each page, not the overall crawl. It's the timeout for the requestHandler function.

https://crawlee.dev/api/3.5/basic-crawler/interface/BasicCrawlerOptions#requestHandlerTimeoutSecs

Add a reply