other-emerald•2y ago

EnqueueLInks

It is possible to add delay's in enqueueLinks? like before to crawl a certain URL I want to add some delay.

5 Replies

You can delay any request in the preNavigationHooks. Just keep in mind that Crawler is scaling concurrency based on the CPU utilization so it will not influence maxConcurrency

Pepa J•2y ago

Hi @Banul; There is basic example to the Lukas's suggestion:

import { utils } from 'crawlee';

// ...

new PlaywrightCrawler({
  // ....
  preNavigationHooks: [
    async (context) => {
        await utils.sleep(10_000);
    },
  ],
  // ...
});

import { utils } from 'crawlee';

// ...

new PlaywrightCrawler({
  // ....
  preNavigationHooks: [
    async (context) => {
        await utils.sleep(10_000);
    },
  ],
  // ...
});

other-emeraldOP•2y ago

hmm does cheerioocrawler has a preNavigationHooks?

Lukas Krivka•2y ago

Yes. You can always find that in the docs https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions

CheerioCrawlerOptions | API | Crawlee

other-emeraldOP•2y ago

Oh I didn't see this. Thanks for this I thought preNavigationHooks option is only for PlaywrightCrawler. Again Thank you !

EnqueueLInks

Did you find this page helpful?