like-gold
like-gold•2y ago

Added "playwright-extra" with "stealthPlugin" and got error "Cannot read properties of undefined"

I have some code using PlaywrightCrawler. I added "playwright-extra" with "stealthPlugin" to this code. Exactly as in documentation [1] I added to my code only this: 
import { firefox } from 'playwright-extra';
import stealthPlugin from 'puppeteer-extra-plugin-stealth';
firefox.use(stealthPlugin());
import { firefox } from 'playwright-extra';
import stealthPlugin from 'puppeteer-extra-plugin-stealth';
firefox.use(stealthPlugin());
The rest of program remains the same as before. And I have useFingerprints: true and launcher: firefox in code. Well, the code works. Bot detection sites report that my crawler has 3 plugins and supports 4 mime types, so something changed. But! I got this is the stdout: 
INFO  PlaywrightCrawler: Starting the crawler.
An error occured while executing "onPageCreated" in plugin "stealth/evasions/user-agent-override": TypeError: Cannot read properties of undefined (reading 'userAgent')
    at Proxy.<anonymous> (.../node_modules/playwright-extra/src/puppeteer-compatiblity-shim/index.ts:217:23)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at processImmediate (node:internal/timers:437:9)
    at process.topLevelDomainCallback (node:domain:161:15)
    at process.callbackTrampoline (node:internal/async_hooks:128:24)
    at async Plugin.onPageCreated (.../node_modules/puppeteer-extra-plugin-stealth/evasions/user-agent-override/index.js:69:8)
INFO  PlaywrightCrawler: Starting the crawler.
An error occured while executing "onPageCreated" in plugin "stealth/evasions/user-agent-override": TypeError: Cannot read properties of undefined (reading 'userAgent')
    at Proxy.<anonymous> (.../node_modules/playwright-extra/src/puppeteer-compatiblity-shim/index.ts:217:23)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at processImmediate (node:internal/timers:437:9)
    at process.topLevelDomainCallback (node:domain:161:15)
    at process.callbackTrampoline (node:internal/async_hooks:128:24)
    at async Plugin.onPageCreated (.../node_modules/puppeteer-extra-plugin-stealth/evasions/user-agent-override/index.js:69:8)
How bad is this? [1] https://crawlee.dev/docs/examples/crawler-plugins
Using puppeteer-extra and playwright-extra | Crawlee
puppeteer-extra and playwright-extra are community-built
8 Replies
like-gold
like-goldOP•2y ago
Is it something related to versions of libraries? here the output of npm ls: 
npm ls
...
├── @apify/tsconfig@0.1.0
├── crawlee@3.6.2
├── playwright-extra@4.3.6
├── playwright@1.39.0
├── puppeteer-extra-plugin-stealth@2.11.2
├── puppeteer-extra@3.3.6
├── puppeteer@21.6.1
├── ts-node@10.9.2
├── ts-promise@2.2.0
└── typescript@4.9.5
npm ls
...
├── @apify/tsconfig@0.1.0
├── crawlee@3.6.2
├── playwright-extra@4.3.6
├── playwright@1.39.0
├── puppeteer-extra-plugin-stealth@2.11.2
├── puppeteer-extra@3.3.6
├── puppeteer@21.6.1
├── ts-node@10.9.2
├── ts-promise@2.2.0
└── typescript@4.9.5
pls help, this exception still there...
Lukas Krivka
Lukas Krivka•2y ago
You are mixing Playwright and Puppeteer, use just one
xenial-black
xenial-black•2y ago
Getting the same error using playwright-extra but doesnt really effect anything, everything continues to work as normal
HonzaS
HonzaS•2y ago
I think puppeteer-extra plugin simply does not work with firefox. As puppeteer is chrome only, this works with playwright - chrome setting. this works without errors 
import { PlaywrightCrawler } from 'crawlee';
import { chromium } from 'playwright-extra';
import stealth from 'puppeteer-extra-plugin-stealth';
import { router } from './routes.js';

await Actor.init();
const proxyConfiguration = await Actor.createProxyConfiguration({groups:['RESIDENTIAL'], countryCode:'US'});
const crawler = new PlaywrightCrawler({
    proxyConfiguration,
    headless:false,
    requestHandler: router,
    launchContext: {
        launcher: chromium.use(stealth())}
});
.....
import { PlaywrightCrawler } from 'crawlee';
import { chromium } from 'playwright-extra';
import stealth from 'puppeteer-extra-plugin-stealth';
import { router } from './routes.js';

await Actor.init();
const proxyConfiguration = await Actor.createProxyConfiguration({groups:['RESIDENTIAL'], countryCode:'US'});
const crawler = new PlaywrightCrawler({
    proxyConfiguration,
    headless:false,
    requestHandler: router,
    launchContext: {
        launcher: chromium.use(stealth())}
});
.....
But sadly still blocked on sites like g2.com or bigspy.com but could help on some other sites.
xenial-black
xenial-black•2y ago
Yup its only effecting firefox https://github.com/berstend/puppeteer-extra/issues/576
GitHub
[Bug] stealth plugin on Firefox throw `TypeError: Cannot read prope...
Steps to reproduce: in a new node project, run PUPPETEER_PRODUCT=firefox npm install puppeteer puppeteer-extra puppeteer-extra-plugin-stealth run the following script: import puppeteer from 'pu...
xenial-black
xenial-black•2y ago
It looks like I have to use the stealth plugin in order to bypass cloudflare captchas using firefox. I thought crawlee already implemented all of the features from the stealth plugin but it wont bypass cloudflare cpatchas on firefox without it. What does the stealth plugin have that crawlee doesnt?
xenial-black
xenial-black•2y ago
I'd also like to know this! I'm fully blocked on Indeed.com regardless of implimentation.
like-gold
like-goldOP•11mo ago
Well (almost a year later, I admit it), I did some clean-up in package.json (no puppeteer any more): 
$ npm ls
...
├── @apify/tsconfig@0.1.0
├── @types/node@20.16.11
├── @types/puppeteer@7.0.4
├── crawlee@3.11.5
├── node-beanstalk@1.6.8
├── playwright-extra@4.3.6
├── playwright@1.48.1
├── puppeteer-extra-plugin-stealth@2.11.2
├── ts-promise@2.2.0
├── tsx@4.19.1
└── typescript@5.6.3
$ npm ls
...
├── @apify/tsconfig@0.1.0
├── @types/node@20.16.11
├── @types/puppeteer@7.0.4
├── crawlee@3.11.5
├── node-beanstalk@1.6.8
├── playwright-extra@4.3.6
├── playwright@1.48.1
├── puppeteer-extra-plugin-stealth@2.11.2
├── ts-promise@2.2.0
├── tsx@4.19.1
└── typescript@5.6.3
And I do this in code: firefox.use(stealthPlugin()); And it does not work, same error as before: 
An error occured while executing "onPageCreated" in plugin "stealth/evasions/user-agent-override": TypeError: Cannot read properties of undefined (reading 'userAgent')
    at Proxy.<anonymous> (.../node_modules/playwright-extra/src/puppeteer-compatiblity-shim/index.ts:217:23)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at process.processImmediate (node:internal/timers:454:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)
    at async Plugin.onPageCreated (.../node_modules/puppeteer-extra-plugin-stealth/evasions/user-agent-override/index.js:69:8)
An error occured while executing "onPageCreated" in plugin "stealth/evasions/user-agent-override": TypeError: Cannot read properties of undefined (reading 'userAgent')
    at Proxy.<anonymous> (.../node_modules/playwright-extra/src/puppeteer-compatiblity-shim/index.ts:217:23)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at process.processImmediate (node:internal/timers:454:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)
    at async Plugin.onPageCreated (.../node_modules/puppeteer-extra-plugin-stealth/evasions/user-agent-override/index.js:69:8)
Just in case all this stuff is still relevant - would be great to fix it. Would you pls look @Lukas Krivka ?

Did you find this page helpful?