PlaywrightCrawler runs into timeout on Apify but works locally
I have a crawler which performs very simple GET requests to a well known marketplace website. One crawler instance only sends 2-3 requests, after that I'm setting up a new crawler with different configuration. Locally, my crawler works just fine, but if I run it on Apify I'm getting the following error for most of the requests:
I only get this behaviour on one specific website, on other websites I'm crawling it works fine. So I'm assuming that I'm running into a blocking mechanism. However, I'm already using residential proxies:
Interesting is that if I mark the session as bad after a timeout and try it again, it seems to work. But this approach is extremely slow and cumbersome, since I have to reset the session after pretty much every request.
Any ideas?
ERROR PlaywrightCrawler: Request failed and reached maximum retries. page.goto: Timeout 10000ms exceeded.I only get this behaviour on one specific website, on other websites I'm crawling it works fine. So I'm assuming that I'm running into a blocking mechanism. However, I'm already using residential proxies:
proxyConfiguration = await Actor.createProxyConfiguration({
groups: ['RESIDENTIAL'],
countryCode: 'DE',
});Interesting is that if I mark the session as bad after a timeout and try it again, it seems to work. But this approach is extremely slow and cumbersome, since I have to reset the session after pretty much every request.
Any ideas?
