The community member is experiencing issues with Cloudflare captchas on a website, where they are immediately prompted with a captcha when accessing the site, even through Google search results. The community members discuss potential solutions, with one suggesting that the website may require captchas for regular users or only for suspicious IPs and fingerprints. Another community member recommends tweaking the scraper's configuration, such as using Playwright with Firefox and a residential proxy, as a potential solution, though they note this is the most expensive option. The community members also provide links to articles that may help with bypassing Cloudflare challenges.
I immediately get captcha on every URL. Accessing it in a normal GUI browser typing site homepage URL: captcha. Searching this site in google, clicking on the link in google results: browser shows site address and...: captcha.
(by the way, they changed it, few months ago this site was not that restrictive)
Well... what is our best solution for sites always showing cloudflare captchas ?
Truly often they always check, even for regular users. However, you can tweak your scraper's configuration to try bypassing it. Generally, using Playwright + firefox with a residential proxy tends to work best, although it’s also the most expensive option.
