Apify Discord Mirror

Updated 13 hours ago

Crawlee Playwright is detected as bot

At a glance

The post discusses an issue where Crawlee Playwright is detected as a bot due to CDP on the browserscan.net website. This is a known issue, also discussed on GitHub. The community members are wondering if Crawlee can come up with a solution.

In the comments, one community member congratulates another on advancing to level 2. Another community member says they are checking with the team and will get back. A community member from the Crawlee team says they are working on a solution that will not use Playwright and should be more unblockable, and provides a link to some tips. Another community member expresses excitement about this news. A community member shares an interesting screenshot and wonders if "Normal" means "Nothing detected", and mentions that the same program is detected on another site. Some community members are curious about the timeline for the new solution.

There is no explicitly marked answer in the comments.

Useful resources
Checking on this page, Crawlee Playwright is detected as bot due to CDP.

https://www.browserscan.net/bot-detection

This is a known issue, also discussed on:

https://github.com/berstend/puppeteer-extra/issues/899

Wondering if Crawlee can come up with a solution?
6
A
S
L
12 comments
@Jeno just advanced to level 2! Thanks for your contributions! πŸŽ‰
checking with team and getting back to you.
Hi @Jeno
We are working on solution that will not use Playwright and should be more unblockable.
Meanwhile you can checkout these tips https://docs.apify.com/academy/anti-scraping#quick-start
That's exciting news!
Very interesting, @Jeno pls look at this screenshot.
As far as i understand: "Normal" = "Nothing detected", right?

Code: Playwright+Crawlee+Firefox+rotating proxies.

And exactly same program is detected on another site, see here:
https://discord.com/channels/801163717915574323/1296398744870457394
Attachment
www.browserscan.detect.png
Can you give a hint at what stage that new solution is? Weeks or months away? Or next major version?
I wanna know that too. It is a great feature .πŸ‘
I wonder, is this an issue with fingerprint-suite or the PlaywrightCrawler implementation?
It's a lot of things. I have read somewhere that CF can detect the synthetic mouse actions by Playwright and Puppeteer. The only solution I had success with was Puppeteer Real Browser. It passes CF easily.
Add a reply
Sign up and join the conversation on Discord