6 replies

How do I delay requests with HttpCrawler?

I am working with an API that has rate-limiting in place. The API gives me a timestamp of when the current rate limit will expire in seconds. I need to delay my next request by this many seconds, which is usually 15 ish minutes.

I tried adding a delay with

setTimeout

setTimeout

and

Promise

Promise

like this and awaiting on it

export function delay(seconds: number): Promise<void> {
    return new Promise((resolve) => setTimeout(resolve, seconds * 1000))
}

export function delay(seconds: number): Promise<void> {
    return new Promise((resolve) => setTimeout(resolve, seconds * 1000))
}

(this delay happens inside my requestHandler)

But when it finishes I get this error on next request

RequestError: read ECONNRESET

RequestError: read ECONNRESET

I am not well-versed with networking, but I think this error is related to it. My understanding is that Crawlee tries to use the same connection that it opened ~15 minutes ago but that connection is closed from the API's side, thus this error.

Any help or suggestions on how to achieve this would be appreciated!

How do I delay requests with HttpCrawler?

Similar Threads