How to aggregate user data from duplicate URLs

How can I save the categories of a product with multiple categories?

Currently, I'm passing the category in

userData

userData

userData

userData to the category handler. However, since a URL is only scraped once, only the first category gets saved and all others discarded.

Here's a minimal example.

import { Router } from "crawlee";

export const router = Router.create();

// ... other handlers

// imagine the category handler is called twice
router.addHandler("CATEGORY", async ({ enqueueLinks }) => {
  // ... parsing

  // result of first call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "foo" },
    label: "DETAIL",
  });

  // result of second call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "bar" },
    label: "DETAIL",
  });
});

router.addHandler("DETAIL", async ({ request }) => {
  // todo: how to get both "foo" and "bar" here?
  const category = request.userData.category;

  // ... saving
});

import { Router } from "crawlee";

export const router = Router.create();

// ... other handlers

// imagine the category handler is called twice
router.addHandler("CATEGORY", async ({ enqueueLinks }) => {
  // ... parsing

  // result of first call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "foo" },
    label: "DETAIL",
  });

  // result of second call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "bar" },
    label: "DETAIL",
  });
});

router.addHandler("DETAIL", async ({ request }) => {
  // todo: how to get both "foo" and "bar" here?
  const category = request.userData.category;

  // ... saving
});

import { Router } from "crawlee";

export const router = Router.create();

// ... other handlers

// imagine the category handler is called twice
router.addHandler("CATEGORY", async ({ enqueueLinks }) => {
  // ... parsing

  // result of first call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "foo" },
    label: "DETAIL",
  });

  // result of second call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "bar" },
    label: "DETAIL",
  });
});

router.addHandler("DETAIL", async ({ request }) => {
  // todo: how to get both "foo" and "bar" here?
  const category = request.userData.category;

  // ... saving
});

import { Router } from "crawlee";

export const router = Router.create();

// ... other handlers

// imagine the category handler is called twice
router.addHandler("CATEGORY", async ({ enqueueLinks }) => {
  // ... parsing

  // result of first call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "foo" },
    label: "DETAIL",
  });

  // result of second call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "bar" },
    label: "DETAIL",
  });
});

router.addHandler("DETAIL", async ({ request }) => {
  // todo: how to get both "foo" and "bar" here?
  const category = request.userData.category;

  // ... saving
});

Pepa J•12/5/25, 10:17 AM

Hi @vwkd ,
If it is not possible to get the category list from the product detail, then you need to make sure, that you will crawl through all the categories first to make your own list of categories per product and then crawl through the product details.

This mostly done by configuring and running 2 crawler Instances - first for the categories and the second one for product details (each crawler instance needs to be configured with its own RequestQueue and own router instance).

During crawling categories you need to track and add the current category to every product, you can use useStateuseState to store the information and load the information withing multiple crawler instances:

import { useState } from 'crawlee';

// initialization categoryRouter, defaultRouter, categoryRequestQueue, detailRequestQueue

categoryRouter.addHandler("CATEGORY", async ({ enqueueLinks, crawler }) => {
  const categoryName = ...;

  const productCategories = await Actor.useState('productCategories', {}); // shared state

  const detailUrls = await page.$$eval('.product-detail a', (linkElements) => linkElements.map((linkEl) => linkEl.href)

  for (const detailUrl of detailUrls) {
    productCategories[detailUrl] ??= [];
    productCategories[detailUrl].push(categoryName);
  }

  // ...add product detail urls to the detailRequestQueue
});

detailRouter.addHandler("DETAIL", async ({ request }) => {
  const productCategories = await Actor.useState('productCategories', {});
  const categories = productCategories[request.url];

  // ... saving
});

import { useState } from 'crawlee';

// initialization categoryRouter, defaultRouter, categoryRequestQueue, detailRequestQueue

categoryRouter.addHandler("CATEGORY", async ({ enqueueLinks, crawler }) => {
  const categoryName = ...;

  const productCategories = await Actor.useState('productCategories', {}); // shared state

  const detailUrls = await page.$$eval('.product-detail a', (linkElements) => linkElements.map((linkEl) => linkEl.href)

  for (const detailUrl of detailUrls) {
    productCategories[detailUrl] ??= [];
    productCategories[detailUrl].push(categoryName);
  }

  // ...add product detail urls to the detailRequestQueue
});

detailRouter.addHandler("DETAIL", async ({ request }) => {
  const productCategories = await Actor.useState('productCategories', {});
  const categories = productCategories[request.url];

  // ... saving
});

Let me know if that makes sense.

import { useState } from 'crawlee'; // initialization categoryRouter, defaultRouter, categoryRequestQueue, detailRequestQueue categoryRouter.addHandler("CATEGORY", async ({ enqueueLinks, crawler }) => { const categoryName = ...; const productCategories = await Actor.useState('productCategories', {}); // shared state const detailUrls = await page.$$eval('.product-detail a', (linkElements) => linkElements.map((linkEl) => linkEl.href) for (const detailUrl of detailUrls) { productCategories[detailUrl] ??= []; productCategories[detailUrl].push(categoryName); } // ...add product detail urls to the detailRequestQueue }); detailRouter.addHandler("DETAIL", async ({ request }) => { const productCategories = await Actor.useState('productCategories', {}); const categories = productCategories[request.url]; // ... saving });