Added "playwright-extra" with "stealthPlugin" and got e...

At a glance

The community member is using PlaywrightCrawler with the "playwright-extra" and "stealthPlugin" libraries to bypass bot detection on websites. They are encountering an error related to the "user-agent-override" plugin, which is causing issues when using the Firefox browser. The community members discuss potential solutions, such as using the Chromium browser instead of Firefox, and note that the "puppeteer-extra" plugin may not work well with Firefox. Some community members suggest that the "stealth" plugin may be necessary to bypass Cloudflare captchas when using Firefox, but it is unclear if there is a definitive solution to the problem.

Useful resources

nnew_in_town

I have some code using PlaywrightCrawler. I added "playwright-extra" with "stealthPlugin" to this code. Exactly as in documentation [1]

I added to my code only this:

Plain Text

import { firefox } from 'playwright-extra';
import stealthPlugin from 'puppeteer-extra-plugin-stealth';
firefox.use(stealthPlugin());

The rest of program remains the same as before. And I have useFingerprints: true and launcher: firefox in code.

Well, the code works. Bot detection sites report that my crawler has 3 plugins and supports 4 mime types, so something changed.
But! I got this is the stdout:

Plain Text

INFO  PlaywrightCrawler: Starting the crawler.
An error occured while executing "onPageCreated" in plugin "stealth/evasions/user-agent-override": TypeError: Cannot read properties of undefined (reading 'userAgent')
    at Proxy.<anonymous> (.../node_modules/playwright-extra/src/puppeteer-compatiblity-shim/index.ts:217:23)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at processImmediate (node:internal/timers:437:9)
    at process.topLevelDomainCallback (node:domain:161:15)
    at process.callbackTrampoline (node:internal/async_hooks:128:24)
    at async Plugin.onPageCreated (.../node_modules/puppeteer-extra-plugin-stealth/evasions/user-agent-override/index.js:69:8)

How bad is this?

[1] https://crawlee.dev/docs/examples/crawler-plugins

8 comments

nnew_in_town

Is it something related to versions of libraries?

here the output of npm ls:

Plain Text

npm ls
...
├── @apify/tsconfig@0.1.0
├── crawlee@3.6.2
├── playwright-extra@4.3.6
├── playwright@1.39.0
├── puppeteer-extra-plugin-stealth@2.11.2
├── puppeteer-extra@3.3.6
├── puppeteer@21.6.1
├── ts-node@10.9.2
├── ts-promise@2.2.0
└── typescript@4.9.5

nnew_in_town

pls help, this exception still there...

LLukas Krivka

You are mixing Playwright and Puppeteer, use just one

DDivMode

Getting the same error using playwright-extra but doesnt really effect anything, everything continues to work as normal

HHonzaS

I think puppeteer-extra plugin simply does not work with firefox. As puppeteer is chrome only, this works with playwright - chrome setting.

this works without errors

Plain Text

import { PlaywrightCrawler } from 'crawlee';
import { chromium } from 'playwright-extra';
import stealth from 'puppeteer-extra-plugin-stealth';
import { router } from './routes.js';

await Actor.init();
const proxyConfiguration = await Actor.createProxyConfiguration({groups:['RESIDENTIAL'], countryCode:'US'});
const crawler = new PlaywrightCrawler({
    proxyConfiguration,
    headless:false,
    requestHandler: router,
    launchContext: {
        launcher: chromium.use(stealth())}
});
.....

But sadly still blocked on sites like g2.com or bigspy.com but could help on some other sites.

DDivMode

Yup its only effecting firefox https://github.com/berstend/puppeteer-extra/issues/576

DDivMode

It looks like I have to use the stealth plugin in order to bypass cloudflare captchas using firefox. I thought crawlee already implemented all of the features from the stealth plugin but it wont bypass cloudflare cpatchas on firefox without it. What does the stealth plugin have that crawlee doesnt?

ddanimalweb

I'd also like to know this! I'm fully blocked on Indeed.com regardless of implimentation.

Add a reply

Apify Discord Mirror

Added "playwright-extra" with "stealthPlugin" and got error "Cannot read properties of undefined"