How do I delay requests with HttpCrawler?

I am working with an API that has rate-limiting in place. The API gives me a timestamp of when the current rate limit will expire in seconds. I need to delay my next request by this many seconds, which is usually 15 ish minutes.

I tried adding a delay with setTimeoutsetTimeout and PromisePromise like this and awaiting on it

export function delay(seconds: number): Promise<void> {
    return new Promise((resolve) => setTimeout(resolve, seconds * 1000))
}

export function delay(seconds: number): Promise<void> {
    return new Promise((resolve) => setTimeout(resolve, seconds * 1000))
}

(this delay happens inside my requestHandler)

But when it finishes I get this error on next request
RequestError: read ECONNRESETRequestError: read ECONNRESET

I am not well-versed with networking, but I think this error is related to it. My understanding is that Crawlee tries to use the same connection that it opened ~15 minutes ago but that connection is closed from the API's side, thus this error.

Any help or suggestions on how to achieve this would be appreciated!

Apify & Crawlee•4y ago•

6 replies

sacred-rose

How do I delay requests with HttpCrawler?

export function delay(seconds: number): Promise<void> {
    return new Promise((resolve) => setTimeout(resolve, seconds * 1000))
}

export function delay(seconds: number): Promise<void> {
    return new Promise((resolve) => setTimeout(resolve, seconds * 1000))
}

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,735 Members

Join

How do I delay requests with HttpCrawler?

How do I delay requests with HttpCrawler?

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads