429

Hello

Does the HTTP crawler use fingerprinting by default when sending requests?
Because I get 429 too many requests from the API I am web scraping, and I suspect that my header values are the cause of it.

azzouzana•1/17/25, 3:52 PM

Yes they are enabled by default. I believe 429 are because of IP address. Try to use proxies

accused-moccasin•1/17/25, 4:49 PM

I prefer residential rotating proxies

Aazzouzana Yes they are enabled by default. I believe 429 are because of IP address. Try to...

fierDeToiMonGrandOP•1/17/25, 6:52 PM

When the request is sent and logged into the request_queues folder the only header i see is

"headers": {
"Content-Type": "application/json",
},

azzouzana•1/17/25, 7:45 PM

Have you tried proxies?

fierDeToiMonGrandOP•1/17/25, 8:45 PM

not yet

MEE6•1/17/25, 8:45 PM

@fierDeToiMonGrand just advanced to level 1! Thanks for your contributions!

Aazzouzana Have you tried proxies?

fierDeToiMonGrandOP•1/17/25, 10:01 PM

tried the proxy and it works well thanks !

azzouzana•1/17/25, 10:06 PM

Great. I'd start with data centers since they're lot cheaper than residential and would probably help with IPs rotation (but if they're antibots is more sophisticated then residential would yield better results.

azzouzana•1/17/25, 10:06 PM

Anyway, glad it helped! Feel free to mark this as solved

fierDeToiMonGrandOP•1/17/25, 10:24 PM

how can i mark it resolved?

azzouzana•1/17/25, 11:13 PM

I just did

Have a nice day/evening/night

fierDeToiMonGrandOP•1/18/25, 12:31 AM

thanks! You too

comprehensive-tomato•1/19/25, 2:09 PM

Yes, you can add a check to switch the proxy when the request status code is 429, which will help resolve the issue.

MEE6•1/19/25, 2:09 PM

@Ван\ just advanced to level 2! Thanks for your contributions!