15 replies

Avoiding Crawler Detection

Hi folks,

I have a crawlee playwright script that I can run locally without being blocked. However, when I deploy and run it on apify I'm getting blocked. Do you have any suggestions on how to avoid it?

Here's the error I'm seeing in the apify console:

WARN  PlaywrightCrawler: Reclaiming failed request back to the list or queue. Navigation timed out after 60 seconds.

WARN PlaywrightCrawler: Reclaiming failed request back to the list or queue. Navigation timed out after 60 seconds.

WARN  PlaywrightCrawler: Reclaiming failed request back to the list or queue. Navigation timed out after 60 seconds.

WARN PlaywrightCrawler: Reclaiming failed request back to the list or queue. Navigation timed out after 60 seconds.

Here's my proxy configuration:

{
    useApifyProxy: true,
    groups: ['RESIDENTIAL'],
    countryCode: 'US'
}

{
    useApifyProxy: true,
    groups: ['RESIDENTIAL'],
    countryCode: 'US'
}

{
    useApifyProxy: true,
    groups: ['RESIDENTIAL'],
    countryCode: 'US'
}

{
    useApifyProxy: true,
    groups: ['RESIDENTIAL'],
    countryCode: 'US'
}

Avoiding Crawler Detection

Similar Threads

Similar Threads

Similar Threads