Delay new requests

I get new requests from third party source and need to sleep crawler before got new request for it. How to realize it?

I tried to create requestQueue, then when I get new request I will add it to queue and run crawler again.

First I create queue and crawler and run it.

const queue = await RequestQueue.open()
const crawler = new PlaywrightCrawler({
  requestQueue: queue,
})
await queue.addRequest(...)
carwler.run()

const queue = await RequestQueue.open()
const crawler = new PlaywrightCrawler({
  requestQueue: queue,
})
await queue.addRequest(...)
carwler.run()

then crawler done work.
All requests from the queue have been processed, the crawler will shut down.All requests from the queue have been processed, the crawler will shut down.

When I got new request I try to add it to queue and run crawler

await queue.addRequest({ url: ..., })
crawlee.run()

await queue.addRequest({ url: ..., })
crawlee.run()

but get message:
All requests from the queue have been processed, the crawler will shut down. All requests from the queue have been processed, the crawler will shut down.

Apify & Crawlee•3y ago•

7 replies

dry-scarlet

Delay new requests

const queue = await RequestQueue.open()
const crawler = new PlaywrightCrawler({
  requestQueue: queue,
})
await queue.addRequest(...)
carwler.run()

const queue = await RequestQueue.open()
const crawler = new PlaywrightCrawler({
  requestQueue: queue,
})
await queue.addRequest(...)
carwler.run()

await queue.addRequest({ url: ..., })
crawlee.run()

await queue.addRequest({ url: ..., })
crawlee.run()

but get message:
All requests from the queue have been processed, the crawler will shut down. All requests from the queue have been processed, the crawler will shut down.

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,735 Members

Join

Delay new requests

Delay new requests

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads