FoudreTower•3mo ago

enqueueLinks with urls don't trigger router handler

Hello my "search" handler enqueues a url ( I have verified and the url exists and is valid ) to my "subprocessors" handler but for some reasons it's not being triggered

router.addHandler(
  "search",
  async ({ request, page, log, pushData, enqueueLinks }) => {
    log.info(`Processing ${request.url} ...`);

    // Find all search result items
    const searchElements = await page
      .locator("div#search div#rso div[data-hveid][lang]")
      .all();

    for (let index = 0; index < 1; index++) {
      const element = searchElements[index];

      const url =
        (await element.locator("a").first().getAttribute("href"))

      await enqueueLinks({
        label: "subprocessors",
        urls: [url],
      });
    }

    console.log(await Dataset.getData());
  }
);

// Not triggered
router.addHandler("subprocessors", async ({ request, page, log, pushData }) => {
  log.info(`Processing ${request.url} ...`);
});

router.addHandler(
  "search",
  async ({ request, page, log, pushData, enqueueLinks }) => {
    log.info(`Processing ${request.url} ...`);

    // Find all search result items
    const searchElements = await page
      .locator("div#search div#rso div[data-hveid][lang]")
      .all();

    for (let index = 0; index < 1; index++) {
      const element = searchElements[index];

      const url =
        (await element.locator("a").first().getAttribute("href"))

      await enqueueLinks({
        label: "subprocessors",
        urls: [url],
      });
    }

    console.log(await Dataset.getData());
  }
);

// Not triggered
router.addHandler("subprocessors", async ({ request, page, log, pushData }) => {
  log.info(`Processing ${request.url} ...`);
});

Solution:

Different domains maybe? Have you tried with a different strategy? Try with All , more information here https://crawlee.dev/js/api/core/enum/EnqueueStrategy

EnqueueStrategy | API | Crawlee for JavaScript · Build reliable cr...

Crawlee helps you build and maintain your crawlers. It's open source, but built by developers who scrape millions of pages every day for a living.

Jump to solution

3 Replies

Solution

azzouzana•3mo ago

Different domains maybe? Have you tried with a different strategy? Try with All , more information here https://crawlee.dev/js/api/core/enum/EnqueueStrategy

EnqueueStrategy | API | Crawlee for JavaScript · Build reliable cr...

Crawlee helps you build and maintain your crawlers. It's open source, but built by developers who scrape millions of pages every day for a living.

FoudreTowerOP•3mo ago

it worked thanks

azzouzana•3mo ago

you're welcome! :perfecto:

enqueueLinks with urls don't trigger router handler

Did you find this page helpful?