7 replies

Delay new requests

I get new requests from third party source and need to sleep crawler before got new request for it. How to realize it?

I tried to create requestQueue, then when I get new request I will add it to queue and run crawler again.

First I create queue and crawler and run it.

const queue = await RequestQueue.open()
const crawler = new PlaywrightCrawler({
  requestQueue: queue,
})
await queue.addRequest(...)
carwler.run()

const queue = await RequestQueue.open()
const crawler = new PlaywrightCrawler({
  requestQueue: queue,
})
await queue.addRequest(...)
carwler.run()

const queue = await RequestQueue.open()
const crawler = new PlaywrightCrawler({
  requestQueue: queue,
})
await queue.addRequest(...)
carwler.run()

const queue = await RequestQueue.open()
const crawler = new PlaywrightCrawler({
  requestQueue: queue,
})
await queue.addRequest(...)
carwler.run()

then crawler done work.

All requests from the queue have been processed, the crawler will shut down.

All requests from the queue have been processed, the crawler will shut down.

All requests from the queue have been processed, the crawler will shut down.

All requests from the queue have been processed, the crawler will shut down.

When I got new request I try to add it to queue and run crawler

await queue.addRequest({ url: ..., })
crawlee.run()

await queue.addRequest({ url: ..., })
crawlee.run()

await queue.addRequest({ url: ..., })
crawlee.run()

await queue.addRequest({ url: ..., })
crawlee.run()

but get message:

 All requests from the queue have been processed, the crawler will shut down.

All requests from the queue have been processed, the crawler will shut down.

 All requests from the queue have been processed, the crawler will shut down.

All requests from the queue have been processed, the crawler will shut down.

Delay new requests

Similar Threads

Similar Threads

Similar Threads