Added "playwright-extra" with "stealthPlugin" and got error "Cannot read properties of undefined"

I have some code using PlaywrightCrawler. I added "playwright-extra" with "stealthPlugin" to this code. Exactly as in documentation [1]

I added to my code only this:

import { firefox } from 'playwright-extra';
import stealthPlugin from 'puppeteer-extra-plugin-stealth';
firefox.use(stealthPlugin());

import { firefox } from 'playwright-extra';
import stealthPlugin from 'puppeteer-extra-plugin-stealth';
firefox.use(stealthPlugin());

The rest of program remains the same as before. And I have

useFingerprints: true

useFingerprints: true

and

launcher: firefox

launcher: firefox

in code.

Well, the code works. Bot detection sites report that my crawler has 3 plugins and supports 4 mime types, so something changed.
But! I got this is the stdout:

INFO  PlaywrightCrawler: Starting the crawler.
An error occured while executing "onPageCreated" in plugin "stealth/evasions/user-agent-override": TypeError: Cannot read properties of undefined (reading 'userAgent')
    at Proxy.<anonymous> (.../node_modules/playwright-extra/src/puppeteer-compatiblity-shim/index.ts:217:23)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at processImmediate (node:internal/timers:437:9)
    at process.topLevelDomainCallback (node:domain:161:15)
    at process.callbackTrampoline (node:internal/async_hooks:128:24)
    at async Plugin.onPageCreated (.../node_modules/puppeteer-extra-plugin-stealth/evasions/user-agent-override/index.js:69:8)

INFO  PlaywrightCrawler: Starting the crawler.
An error occured while executing "onPageCreated" in plugin "stealth/evasions/user-agent-override": TypeError: Cannot read properties of undefined (reading 'userAgent')
    at Proxy.<anonymous> (.../node_modules/playwright-extra/src/puppeteer-compatiblity-shim/index.ts:217:23)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at processImmediate (node:internal/timers:437:9)
    at process.topLevelDomainCallback (node:domain:161:15)
    at process.callbackTrampoline (node:internal/async_hooks:128:24)
    at async Plugin.onPageCreated (.../node_modules/puppeteer-extra-plugin-stealth/evasions/user-agent-override/index.js:69:8)

How bad is this?

[1] https://crawlee.dev/docs/examples/crawler-plugins

Using puppeteer-extra and playwright-extra | Crawlee

puppeteer-extra and playwright-extra are community-built

sacred-emeraldOP•12/21/23, 1:21 PM

Is it something related to versions of libraries?

here the output of npm ls:

sacred-emeraldOP•1/2/24, 3:02 PM

pls help, this exception still there...

Lukas Krivka•1/12/24, 4:39 PM

You are mixing Playwright and Puppeteer, use just one

progressive-amaranth•2/2/24, 12:04 PM

Getting the same error using playwright-extra but doesnt really effect anything, everything continues to work as normal

HonzaS•2/6/24, 10:40 AM

I think puppeteer-extra plugin simply does not work with firefox. As puppeteer is chrome only, this works with playwright - chrome setting.

this works without errors

But sadly still blocked on sites like g2.com or bigspy.com but could help on some other sites.

progressive-amaranth•2/6/24, 8:37 PM

Yup its only effecting firefox https://github.com/berstend/puppeteer-extra/issues/576

GitHub

[Bug] stealth plugin on Firefox throw `TypeError: Cannot read prope...

Steps to reproduce: in a new node project, run PUPPETEER_PRODUCT=firefox npm install puppeteer puppeteer-extra puppeteer-extra-plugin-stealth run the following script: import puppeteer from 'pu...

progressive-amaranth•2/8/24, 9:47 PM

It looks like I have to use the stealth plugin in order to bypass cloudflare captchas using firefox. I thought crawlee already implemented all of the features from the stealth plugin but it wont bypass cloudflare cpatchas on firefox without it. What does the stealth plugin have that crawlee doesnt?

popular-magenta•2/12/24, 6:20 PM

I'd also like to know this! I'm fully blocked on Indeed.com regardless of implimentation.

sacred-emeraldOP•10/17/24, 7:19 PM

Well (almost a year later, I admit it), I did some clean-up in package.json (no puppeteer any more):

And I do this in code:

And it does not work, same error as before:

Just in case all this stuff is still relevant - would be great to fix it. Would you pls look @Lukas Krivka ?