hi, thank you for the tool

hi, thank you for the tool PlaywrightCrawler. I want to ask How to handle 429 status code caused by requesting too often? Is there a sleep-for-a-few-seconds method to handle this? Thanks for the attention.

Pepa J•4/18/24, 6:23 AM

Hi @jackfsuia ,
Great question! Please consider using #crawlee-js #

apify-python channels next time, so we may provide you better answer based on the tools, that you are using.

For Crawlee, there is a paramter to limit the amaount of requests in minute:

You migh also find helpful article about how to deal with rate-limiting by using proxies https://docs.apify.com/academy/anti-scraping/techniques/rate-limiting

Rate-limiting | Academy | Apify Documentation

Learn about rate-limiting, a common tactic used by websites to avoid a large and non-human rate of requests coming from a single IP address.