5 replies

enqueueLinks with urls don't trigger router handler

Hello my "search" handler enqueues a url ( I have verified and the url exists and is valid ) to my "subprocessors" handler but for some reasons it's not being triggered

router.addHandler(
  "search",
  async ({ request, page, log, pushData, enqueueLinks }) => {
    log.info(`Processing ${request.url} ...`);

    // Find all search result items
    const searchElements = await page
      .locator("div#search div#rso div[data-hveid][lang]")
      .all();

    for (let index = 0; index < 1; index++) {
      const element = searchElements[index];

      const url =
        (await element.locator("a").first().getAttribute("href"))

      await enqueueLinks({
        label: "subprocessors",
        urls: [url],
      });
    }

    console.log(await Dataset.getData());
  }
);

// Not triggered
router.addHandler("subprocessors", async ({ request, page, log, pushData }) => {
  log.info(`Processing ${request.url} ...`);
});

router.addHandler(
  "search",
  async ({ request, page, log, pushData, enqueueLinks }) => {
    log.info(`Processing ${request.url} ...`);

    // Find all search result items
    const searchElements = await page
      .locator("div#search div#rso div[data-hveid][lang]")
      .all();

    for (let index = 0; index < 1; index++) {
      const element = searchElements[index];

      const url =
        (await element.locator("a").first().getAttribute("href"))

      await enqueueLinks({
        label: "subprocessors",
        urls: [url],
      });
    }

    console.log(await Dataset.getData());
  }
);

// Not triggered
router.addHandler("subprocessors", async ({ request, page, log, pushData }) => {
  log.info(`Processing ${request.url} ...`);
});

Solution

Different domains maybe? Have you tried with a different strategystrategy? Try with AllAll , more information here https://crawlee.dev/js/api/core/enum/EnqueueStrategy

EnqueueStrategy | API | Crawlee for JavaScript · Build reliable cr...

Crawlee helps you build and maintain your crawlers. It's open source, but built by developers who scrape millions of pages every day for a living.

Jump to solution

Apify & Crawlee•13mo ago•

5 replies

FoudreTower

enqueueLinks with urls don't trigger router handler

Hello my "search" handler enqueues a url ( I have verified and the url exists and is valid ) to my "subprocessors" handler but for some reasons it's not being triggered

router.addHandler(
  "search",
  async ({ request, page, log, pushData, enqueueLinks }) => {
    log.info(`Processing ${request.url} ...`);

    // Find all search result items
    const searchElements = await page
      .locator("div#search div#rso div[data-hveid][lang]")
      .all();

    for (let index = 0; index < 1; index++) {
      const element = searchElements[index];

      const url =
        (await element.locator("a").first().getAttribute("href"))

      await enqueueLinks({
        label: "subprocessors",
        urls: [url],
      });
    }

    console.log(await Dataset.getData());
  }
);

// Not triggered
router.addHandler("subprocessors", async ({ request, page, log, pushData }) => {
  log.info(`Processing ${request.url} ...`);
});

router.addHandler(
  "search",
  async ({ request, page, log, pushData, enqueueLinks }) => {
    log.info(`Processing ${request.url} ...`);

    // Find all search result items
    const searchElements = await page
      .locator("div#search div#rso div[data-hveid][lang]")
      .all();

    for (let index = 0; index < 1; index++) {
      const element = searchElements[index];

      const url =
        (await element.locator("a").first().getAttribute("href"))

      await enqueueLinks({
        label: "subprocessors",
        urls: [url],
      });
    }

    console.log(await Dataset.getData());
  }
);

// Not triggered
router.addHandler("subprocessors", async ({ request, page, log, pushData }) => {
  log.info(`Processing ${request.url} ...`);
});

Solution

Different domains maybe? Have you tried with a different strategystrategy? Try with AllAll , more information here https://crawlee.dev/js/api/core/enum/EnqueueStrategy

EnqueueStrategy | API | Crawlee for JavaScript · Build reliable cr...

Crawlee helps you build and maintain your crawlers. It's open source, but built by developers who scrape millions of pages every day for a living.

Jump to solution

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

15,108 Members

Join

enqueueLinks with urls don't trigger router handler

enqueueLinks with urls don't trigger router handler

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads