Running crawlee multiple times with the same URL

Hi!

I am trying to build a crawler using PuppeteerCrawler. The crawler will be started by sending a POST to an API endpoint. The API is implemented using azure durable functions.

The first time I call the API it works as expected. The next time I call it I get no output. This is the log output on the second run:

INFO  PuppeteerCrawler: Initializing the crawler.
INFO  PuppeteerCrawler: All requests from the queue have been processed, the crawler will shut down.

INFO  PuppeteerCrawler: Initializing the crawler.
INFO  PuppeteerCrawler: All requests from the queue have been processed, the crawler will shut down.

How do I configure crawlee such that every call to the API runs a new crawl?

Here is my current implementation. This function is called from an orchestrator function.

const activityFunction: AzureFunction = async function (
  context: Context,
  crawlerParameters: CrawlingParameters
) {
  process.env.CRAWLEE_STORAGE_DIR = os.tmpdir();

  const nettsider = new Set();
  const crawler = new PuppeteerCrawler({
    async requestHandler({ request, page, enqueueLinks }) {
      nettsider.add({ title: await page.title(), url: request.url });
      await enqueueLinks({ exclude: [/.pdf$/, /.doc$/] });
    },
    maxRequestsPerCrawl: crawlerParameters.maxLenker,
  });
  await crawler.run([crawlerParameters.startUrl]);

  return Array.from(nettsider);
};

const activityFunction: AzureFunction = async function (
  context: Context,
  crawlerParameters: CrawlingParameters
) {
  process.env.CRAWLEE_STORAGE_DIR = os.tmpdir();

  const nettsider = new Set();
  const crawler = new PuppeteerCrawler({
    async requestHandler({ request, page, enqueueLinks }) {
      nettsider.add({ title: await page.title(), url: request.url });
      await enqueueLinks({ exclude: [/.pdf$/, /.doc$/] });
    },
    maxRequestsPerCrawl: crawlerParameters.maxLenker,
  });
  await crawler.run([crawlerParameters.startUrl]);

  return Array.from(nettsider);
};

Apify & Crawlee•3y ago•

3 replies

progressive-amaranth

Running crawlee multiple times with the same URL

INFO  PuppeteerCrawler: Initializing the crawler.
INFO  PuppeteerCrawler: All requests from the queue have been processed, the crawler will shut down.

INFO  PuppeteerCrawler: Initializing the crawler.
INFO  PuppeteerCrawler: All requests from the queue have been processed, the crawler will shut down.

How do I configure crawlee such that every call to the API runs a new crawl?

Here is my current implementation. This function is called from an orchestrator function.

const activityFunction: AzureFunction = async function (
  context: Context,
  crawlerParameters: CrawlingParameters
) {
  process.env.CRAWLEE_STORAGE_DIR = os.tmpdir();

  const nettsider = new Set();
  const crawler = new PuppeteerCrawler({
    async requestHandler({ request, page, enqueueLinks }) {
      nettsider.add({ title: await page.title(), url: request.url });
      await enqueueLinks({ exclude: [/.pdf$/, /.doc$/] });
    },
    maxRequestsPerCrawl: crawlerParameters.maxLenker,
  });
  await crawler.run([crawlerParameters.startUrl]);

  return Array.from(nettsider);
};

const activityFunction: AzureFunction = async function (
  context: Context,
  crawlerParameters: CrawlingParameters
) {
  process.env.CRAWLEE_STORAGE_DIR = os.tmpdir();

  const nettsider = new Set();
  const crawler = new PuppeteerCrawler({
    async requestHandler({ request, page, enqueueLinks }) {
      nettsider.add({ title: await page.title(), url: request.url });
      await enqueueLinks({ exclude: [/.pdf$/, /.doc$/] });
    },
    maxRequestsPerCrawl: crawlerParameters.maxLenker,
  });
  await crawler.run([crawlerParameters.startUrl]);

  return Array.from(nettsider);
};

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,290 Members

Join

Running crawlee multiple times with the same URL

Running crawlee multiple times with the same URL

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads