exotic-emerald•11mo ago
Crawlee Playwright is detected as bot
Checking on this page, Crawlee Playwright is detected as bot due to CDP.
https://www.browserscan.net/bot-detection
This is a known issue, also discussed on:
https://github.com/berstend/puppeteer-extra/issues/899
Wondering if Crawlee can come up with a solution?
BrowserScan
BrowserScan - Robot Detection/WebDriver
Bot Test, WebDriver Test, Discord bots, Cloudflare Turnstile, Google reCAPTCHA, gives you a powerful tool to prevent online fraud
GitHub
[Bug] Stealth being detected by Chrome DevTools Protocol (CDP) · Is...
Puppeeteer stealth is now being easily detected, checkout https://deviceandbrowserinfo.com/learning_zone/articles/detecting-headless-chrome-puppeteer-2024
16 Replies
@Jeno just advanced to level 2! Thanks for your contributions! 🎉
View post on community site
This post has been pushed to the community knowledgebase. Any replies in this thread will be synced to the community site.
Apify Community
checking with team and getting back to you.
Hi @Jeno
We are working on solution that will not use Playwright and should be more unblockable.
Meanwhile you can checkout these tips https://docs.apify.com/academy/anti-scraping#quick-start
Anti-scraping protections | Academy | Apify Documentation
Understand the various anti-scraping measures different sites use to prevent bots from accessing them, and how to appear more human to fix these issues.
exotic-emeraldOP•11mo ago
That's exciting news!
passive-yellow•11mo ago
Very interesting, @Jeno pls look at this screenshot.
As far as i understand: "Normal" = "Nothing detected", right?
Code: Playwright+Crawlee+Firefox+rotating proxies.
And exactly same program is detected on another site, see here:
https://discord.com/channels/801163717915574323/1296398744870457394
exotic-emeraldOP•10mo ago
Can you give a hint at what stage that new solution is? Weeks or months away? Or next major version?
I wanna know that too. It is a great feature .👏
👀
eager-peach•9mo ago
👀
👀
xenial-black•9mo ago
GitHub
GitHub - apify/fingerprint-suite: Browser fingerprinting tools for ...
Browser fingerprinting tools for anonymizing your scrapers. Developed by Apify. - apify/fingerprint-suite
exotic-emeraldOP•8mo ago
It's a lot of things. I have read somewhere that CF can detect the synthetic mouse actions by Playwright and Puppeteer. The only solution I had success with was Puppeteer Real Browser. It passes CF easily.
How can we use real browsers at scale?
exotic-emeraldOP•6mo ago
Crawlee silently added Camoufox and it's amazing! Real browser was a pain to use. I am using Camoufox with simple datacenter proxies and easily passing everything. It is available as a template. Make sure you use socks5 to avoid some edge cases.
haha its still in beta 🙂 we are about to post it on social media this week! glad you liked it 😄