429
Hello π
Does the HTTP crawler use fingerprinting by default when sending requests?
Because I get 429 too many requests from the API I am web scraping, and I suspect that my header values are the cause of it.
13 Replies
Yes they are enabled by default. I believe 429 are because of IP address. Try to use proxies
metropolitan-bronzeβ’8mo ago
I prefer residential rotating proxies
When the request is sent and logged into the request_queues folder the only header i see is
"headers": {
"Content-Type": "application/json",
},
Have you tried proxies?
not yet
@fierDeToiMonGrand just advanced to level 1! Thanks for your contributions! π
tried the proxy and it works well thanks !
Great. I'd start with data centers since they're lot cheaper than residential and would probably help with IPs rotation (but if they're antibots is more sophisticated then residential would yield better results.
Anyway, glad it helped! Feel free to mark this as solved βΊοΈ
how can i mark it resolved?
I just did β
Have a nice day/evening/night βΊοΈ
thanks! You too π
wise-whiteβ’8mo ago
Yes, you can add a check to switch the proxy when the request status code is 429, which will help resolve the issue.
@ΠΠ°Π½\ just advanced to level 2! Thanks for your contributions! π