How to recrawl initial page for new links without purging to keep track of duplicates?

Thank you for releasing Crawlee as an open source project.

I have setup a very simple CheerioCrawler that crawls the home page of a news website for starters.
I am then scraping certain articles and saving some information to an external database. I aim to run the crawler at a regular interval (every few hours) to check for new links/articles.
I want to keep track of what I've crawled in previous runs as to not re-visit those pages and waste resources (mine and the hosts), so I set CRAWLEE_PURGE_ON_START to false to keep track of what's been crawled.

Current State:
- Once I run the crawler once, the home page is marked as "handled" and never visited again on subsequent runs to look for new links within it.
Desired State:
- On each new run, crawl the same home page, and enqueue any and only new links found for handling/scaping.

Is there a way to make my starting home page (example.com) re-crawlable on each run without purging? I believe it's something I can add within the default handler, I'm just not sure what exactly it is.

// .env
CRAWLEE_PURGE_ON_START=false

// main.ts
const startUrls = ["example.com"]
const crawler = new CheerioCrawler({
  requestHandler: router,
  maxRequestsPerCrawl: 5,
  maxConcurrency: 1,
  // maxRequestsPerMinute: 30,
});
const main = async () => {
  await crawler.addRequests(startUrls);
  await crawler.run();
};
main();

// router.ts
router.addDefaultHandler(async ({ enqueueLinks, log }) => {
  log.info(`enqueueing new URLs`);
  await enqueueLinks({
    strategy: EnqueueStrategy.SameHostname,
    globs: [
      "https://example.com/news/*",
    ],
    label: "detail"
  })
});

router.addHandler("detail", async ({ request: req, $, log }) => {
  const title = await page.title();
  log.info(`Title of ${request.loadedUrl} is '${title}'`);

  // Save to DB Code here
});

// .env
CRAWLEE_PURGE_ON_START=false

// main.ts
const startUrls = ["example.com"]
const crawler = new CheerioCrawler({
  requestHandler: router,
  maxRequestsPerCrawl: 5,
  maxConcurrency: 1,
  // maxRequestsPerMinute: 30,
});
const main = async () => {
  await crawler.addRequests(startUrls);
  await crawler.run();
};
main();

// router.ts
router.addDefaultHandler(async ({ enqueueLinks, log }) => {
  log.info(`enqueueing new URLs`);
  await enqueueLinks({
    strategy: EnqueueStrategy.SameHostname,
    globs: [
      "https://example.com/news/*",
    ],
    label: "detail"
  })
});

router.addHandler("detail", async ({ request: req, $, log }) => {
  const title = await page.title();
  log.info(`Title of ${request.loadedUrl} is '${title}'`);

  // Save to DB Code here
});

Thank you.

// .env
CRAWLEE_PURGE_ON_START=false

// main.ts
const startUrls = ["example.com"]
const crawler = new CheerioCrawler({
  requestHandler: router,
  maxRequestsPerCrawl: 5,
  maxConcurrency: 1,
  // maxRequestsPerMinute: 30,
});
const main = async () => {
  await crawler.addRequests(startUrls);
  await crawler.run();
};
main();

// router.ts
router.addDefaultHandler(async ({ enqueueLinks, log }) => {
  log.info(`enqueueing new URLs`);
  await enqueueLinks({
    strategy: EnqueueStrategy.SameHostname,
    globs: [
      "https://example.com/news/*",
    ],
    label: "detail"
  })
});

router.addHandler("detail", async ({ request: req, $, log }) => {
  const title = await page.title();
  log.info(`Title of ${request.loadedUrl} is '${title}'`);

  // Save to DB Code here
});

// .env
CRAWLEE_PURGE_ON_START=false

// main.ts
const startUrls = ["example.com"]
const crawler = new CheerioCrawler({
  requestHandler: router,
  maxRequestsPerCrawl: 5,
  maxConcurrency: 1,
  // maxRequestsPerMinute: 30,
});
const main = async () => {
  await crawler.addRequests(startUrls);
  await crawler.run();
};
main();

// router.ts
router.addDefaultHandler(async ({ enqueueLinks, log }) => {
  log.info(`enqueueing new URLs`);
  await enqueueLinks({
    strategy: EnqueueStrategy.SameHostname,
    globs: [
      "https://example.com/news/*",
    ],
    label: "detail"
  })
});

router.addHandler("detail", async ({ request: req, $, log }) => {
  const title = await page.title();
  log.info(`Title of ${request.loadedUrl} is '${title}'`);

  // Save to DB Code here
});

Thank you.

How to recrawl initial page for new links without purging to keep track of duplicates?

How to recrawl initial page for new links without purging to keep track of duplicates?

Similar Threads

Similar Threads

Similar Threads