How to stop the crawl after a certain time?

I use crawlee to warm up my caches. And I set a stop after 5000 pages. But sometimes it takes more or less time to crawl them. I prefer to tell it to stop the crawl after x minutes. Is it possible via the configuration ?

rubber-blueOP•5/10/23, 1:01 PM

For the moment I did:

const timeout = (ms: number) =>
  new Promise<void>((resolve) => {
    setTimeout(() => {
      console.log(`End after ${configuration.CRAWL_TIMEOUT} secondes`);
      resolve();
    }, ms);
  });

const run = async () => {
  const promesseCrawler = crawler.run([configuration.START_URL]);
  const promesseTimeout = timeout(configuration.CRAWL_TIMEOUT * 1000);
  await Promise.race([promesseCrawler, promesseTimeout]);
  process.exit(0);
};

const timeout = (ms: number) =>
  new Promise<void>((resolve) => {
    setTimeout(() => {
      console.log(`End after ${configuration.CRAWL_TIMEOUT} secondes`);
      resolve();
    }, ms);
  });

const run = async () => {
  const promesseCrawler = crawler.run([configuration.START_URL]);
  const promesseTimeout = timeout(configuration.CRAWL_TIMEOUT * 1000);
  await Promise.race([promesseCrawler, promesseTimeout]);
  process.exit(0);
};

But I think you have a configuration for that?

Oleg V.•5/10/23, 1:38 PM

You can try:

You can use method abort() of AutoscaledPool class: https://crawlee.dev/api/next/core/class/AutoscaledPool#abort
Or you can set maxRequestsPerCrawlhttps://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#maxRequestsPerCrawl