bot detection (captcha) changed, Playwright+Crawlee+Firefox+rotating proxies does not help any more
I have a program: Playwright+Crawlee+Firefox+rotating proxies used to scrape jobs from wellfound.com In may 2024 (and earlier) it worked quite well, many months, despite captcha protection on site.
Today I get HTTP 403 and captcha (from ct.captcha-delivery.com). My code is not changed!
Proxies: iproyal.com, "residential-proxies", session time 1 min ("sticky session"). What I did: in the same session accessed URL1 and than URL2. URL1 has no captcha, URL2 contains info I need, and is/was protected with captcha. In the past the trick with "URL1 and than URL2 in the same session" worked well. Today I get captcha when accessing URL2.
What I tried: switched between Chrome and Firefox in my code. For Chrome tried with
Still see that captcha. Tried to access the site with normal GUI browser (Firefox) through iproyal.com "sticky session": accessing URL1 and than URL2: no captcha.
It means: proxies are still OK, they are not detected!
Bottom line: something changed, bot detection improved.
What is our answer?
Is it something similar to this: Crawlee Playwright is detected as bot
@Jeno what solution you found?
Today I get HTTP 403 and captcha (from ct.captcha-delivery.com). My code is not changed!
Proxies: iproyal.com, "residential-proxies", session time 1 min ("sticky session"). What I did: in the same session accessed URL1 and than URL2. URL1 has no captcha, URL2 contains info I need, and is/was protected with captcha. In the past the trick with "URL1 and than URL2 in the same session" worked well. Today I get captcha when accessing URL2.
What I tried: switched between Chrome and Firefox in my code. For Chrome tried with
chromium.use(stealthPlugin()) and without it.Still see that captcha. Tried to access the site with normal GUI browser (Firefox) through iproyal.com "sticky session": accessing URL1 and than URL2: no captcha.
It means: proxies are still OK, they are not detected!
Bottom line: something changed, bot detection improved.
What is our answer?
Is it something similar to this: Crawlee Playwright is detected as bot
@Jeno what solution you found?
