Avoiding Crawler Detection

Hi folks,

I have a crawlee playwright script that I can run locally without being blocked. However, when I deploy and run it on apify I'm getting blocked. Do you have any suggestions on how to avoid it?

Here's the error I'm seeing in the apify console: WARN PlaywrightCrawler: Reclaiming failed request back to the list or queue. Navigation timed out after 60 seconds.WARN PlaywrightCrawler: Reclaiming failed request back to the list or queue. Navigation timed out after 60 seconds.

Here's my proxy configuration:

{
    useApifyProxy: true,
    groups: ['RESIDENTIAL'],
    countryCode: 'US'
}

{
    useApifyProxy: true,
    groups: ['RESIDENTIAL'],
    countryCode: 'US'
}

Apify & Crawlee•14mo ago•

15 replies

zac

Avoiding Crawler Detection

{
    useApifyProxy: true,
    groups: ['RESIDENTIAL'],
    countryCode: 'US'
}

{
    useApifyProxy: true,
    groups: ['RESIDENTIAL'],
    countryCode: 'US'
}

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

15,894 Members

Join

Avoiding Crawler Detection

Avoiding Crawler Detection

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads