1 reply

How to aggregate user data from duplicate URLs

🎭PlaywrightCrawler👨‍💻Web-Scraping🚀RequestQueue

How can I save the categories of a product with multiple categories?

Currently, I'm passing the category in userDatauserData to the category handler. However, since a URL is only scraped once, only the first category gets saved and all others discarded.

Here's a minimal example.

import { Router } from "crawlee";

export const router = Router.create();

// ... other handlers

// imagine the category handler is called twice
router.addHandler("CATEGORY", async ({ enqueueLinks }) => {
  // ... parsing

  // result of first call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "foo" },
    label: "DETAIL",
  });

  // result of second call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "bar" },
    label: "DETAIL",
  });
});

router.addHandler("DETAIL", async ({ request }) => {
  // todo: how to get both "foo" and "bar" here?
  const category = request.userData.category;

  // ... saving
});

import { Router } from "crawlee";

export const router = Router.create();

// ... other handlers

// imagine the category handler is called twice
router.addHandler("CATEGORY", async ({ enqueueLinks }) => {
  // ... parsing

  // result of first call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "foo" },
    label: "DETAIL",
  });

  // result of second call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "bar" },
    label: "DETAIL",
  });
});

router.addHandler("DETAIL", async ({ request }) => {
  // todo: how to get both "foo" and "bar" here?
  const category = request.userData.category;

  // ... saving
});

Apify & Crawlee•6mo ago•

1 reply

vwkd

How to aggregate user data from duplicate URLs

🎭PlaywrightCrawler👨‍💻Web-Scraping🚀RequestQueue

import { Router } from "crawlee";

export const router = Router.create();

// ... other handlers

// imagine the category handler is called twice
router.addHandler("CATEGORY", async ({ enqueueLinks }) => {
  // ... parsing

  // result of first call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "foo" },
    label: "DETAIL",
  });

  // result of second call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "bar" },
    label: "DETAIL",
  });
});

router.addHandler("DETAIL", async ({ request }) => {
  // todo: how to get both "foo" and "bar" here?
  const category = request.userData.category;

  // ... saving
});

import { Router } from "crawlee";

export const router = Router.create();

// ... other handlers

// imagine the category handler is called twice
router.addHandler("CATEGORY", async ({ enqueueLinks }) => {
  // ... parsing

  // result of first call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "foo" },
    label: "DETAIL",
  });

  // result of second call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "bar" },
    label: "DETAIL",
  });
});

router.addHandler("DETAIL", async ({ request }) => {
  // todo: how to get both "foo" and "bar" here?
  const category = request.userData.category;

  // ... saving
});

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,851 Members

Join

How to aggregate user data from duplicate URLs

How to aggregate user data from duplicate URLs

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads