web Crawler got blocked in selenium

message like Verifying you are human. This may take a few seconds.
In robots.txt
User-agent: *
Disallow: /
please share your suggestion on above problem

memo23•5/6/24, 7:35 AM

@Umang dm me

lemurio•5/8/24, 9:02 AM

hi, try to check out this article for more info https://docs.apify.com/academy/anti-scraping

Anti-scraping protections | Academy | Apify Documentation

Understand the various anti-scraping measures different sites use to prevent bots from accessing them, and how to appear more human to fix these issues.