molecular-blue•3y ago
Cloudflare bypass fingerprints
Dear all, I am currently experimenting to bypass cloudflare Security. I am using playwright but getting dectected. My ip is whitelisted. Can any one of you please guide me how can I create a fingerprint which let me bypass cloudflare ?
Thankyou in advance.
10 Replies
sensitive-blue•3y ago
use proxies? 🤔
molecular-blueOP•3y ago
But if my ip is already whitelisted then what difference a proxy could make ?
In prod I will for sure use proxies but for testing am using my residential internet ip.
fair-rose•3y ago
Are you using just playwright or crawlee playwright crawler? Crawlers already have fingerprints enabled by default. But may also want to try full (non-headless mode) or playwright + firefox combination
molecular-blueOP•3y ago
Yes am using crawlee playwright. What configuration I need to change to run in full mode and with firfox ? Any link to docs ?
fair-rose•3y ago
firefox: https://crawlee.dev/docs/examples/playwright-crawler-firefox
headless (should be set to false): https://crawlee.dev/api/playwright-crawler/interface/PlaywrightCrawlerOptions#headless
molecular-blueOP•3y ago
Thanks for the links. I think know the problem. Crawlee doesnt let the cloudflare complete the bot detection. It launches the browser, and before even cloudflare could check it, crawlee just close the browser.
Anyway to avoid this behaviour ?
@curioussoul just advanced to level 1! Thanks for your contributions! 🎉
fair-rose•3y ago
oh yesss, I completely forgot about it. it should be addressed in one of the next releases - there apparently was some change in how cloudflare works. For now the fastest way to solve it would be to add
sessionPoolOptions: { blockedStatusCodes: [] } to your crawler optionsmolecular-blueOP•3y ago
Thanks for the help. It worked now. Its bypassing CF secuirty in headless mode as well now.
Great work guys. I get to know about crawlee few days ago and today I tried it and its amazing. Lots of respect for you guys.
fair-rose•3y ago
Thanks for the feedback 👌 Trying our best 🙂