#crawlee-js - Apify & Crawlee

How to aggregate user data from duplicate URLs

vwkd12/2/25, 1:25 AM

How can I save the categories of a product with multiple categories?

Currently, I'm passing the category in

userData

userData

to the category handler. However, since a URL is only scraped once, only the first category gets saved and all others discarded.

Here's a minimal example.

import { Router } from "crawlee";

export const router = Router.create();

// ... other handlers

// imagine the category handler is called twice
router.addHandler("CATEGORY", async ({ enqueueLinks }) => {
  // ... parsing

  // result of first call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "foo" },
    label: "DETAIL",
  });

  // result of second call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "bar" },
    label: "DETAIL",
  });
});

router.addHandler("DETAIL", async ({ request }) => {
  // todo: how to get both "foo" and "bar" here?
  const category = request.userData.category;

  // ... saving
});

import { Router } from "crawlee";

export const router = Router.create();

// ... other handlers

// imagine the category handler is called twice
router.addHandler("CATEGORY", async ({ enqueueLinks }) => {
  // ... parsing

  // result of first call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "foo" },
    label: "DETAIL",
  });

  // result of second call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "bar" },
    label: "DETAIL",
  });
});

router.addHandler("DETAIL", async ({ request }) => {
  // todo: how to get both "foo" and "bar" here?
  const category = request.userData.category;

  // ... saving
});

Impit part of Crawlee

Louis Deconinck12/1/25, 9:37 PM

Will Impit become a part of Crawlee and replace GotScraping?

AI | Full-Stack Engineer

JoyfulKiwi55511/28/25, 6:00 PM

Hello, everyone! Nice to meet you all.
Now I am looking for a work and if you have any idea, I can bring it to life!
https://daichi555-portfolio.netlify.app/

Daichi Yamamoto - Full Stack Developer

AI & Automation Engineer and Full Stack Developer with 7+ years of experience

How to use PlaywrightCrawler to listen for ChatGPT's SSE responses on a web page.

fourier11/24/25, 11:38 AM

pls. I need help.

Error handling

Michael11/13/25, 7:57 AM

Hello error handling behaviour is little unclear for some requests 400 are considered sometime as ok sometime as error even more weird i sometime got empty html (i'm 100% sure that not the produced response)

<html><head></head><body></body></html>

<html><head></head><body></body></html>

? even after adding

await waitForSelector('#myID')

await waitForSelector('#myID')

how i suppose to deal with that ? thanks

Retry requests with different headers and etc

memo2311/11/25, 11:43 PM

Can we get this page also in JS version: https://crawlee.dev/python/docs/guides/error-handling
Also I am interested regarding best practices how to change headers when 403 or 429 are encountered, so I don't repeat same request with same headers and different IP only

Error handling | Crawlee for Python · Fast, reliable Python web cr...

Crawlee helps you build and maintain your Python crawlers. It's open source and modern, with type hints for Python to help you catch bugs early.

Get A Job Offer

Original message was deleted

Impit client: how to set http2, and how can I order headers & pseudo headers chirurgically

azzouzana10/25/25, 10:57 PM

Basically the title

Problem with scraping a site that requires login

Casper10/22/25, 9:46 PM

I have a paid actor I am renting out to customers that is failing because of a recent anti bot mitigation that prevents scraping pages past 10 without logging in. I have implemented Google login and store session cookies in a shared key value store for the actor to use and this seem to work fine. However Google has flagged account and logins as being a bot and has since terminated the account thus login fails and then scraping fails as well. Before the Google account termination, I experienced that the site I scrape, also seemed to throttle my requests - however this is without using a proxy so might be possible to circumvent, however this has never been an issue before with this site.

The site has option for Google, Facebook, Apple or email login and I chose Google because email requires to receive a login code to the email each time a login is performed, which I couldn't automate.

I have been trying to resolve this for the past week and was successfull until the Google login termination.

I am using Crawlee Playwright and run only 1 concurrent browser context to not overwhelm or batch requests against the site.

Do you have experience with how to deal with such anti bot measures reliably?

How would you log a nicely formatted full JSON response for debugging?

Louis Deconinck10/21/25, 9:27 PM

How would you log a nicely formatted full JSON response for debugging using the log?

How to use Impit as the http client for Crawlee?

Louis Deconinck10/21/25, 8:19 PM

How can I use Impit as the http client when using the httpcrawler?

Why does crawlee generate warning when both blockedStatusCodes and retryOnBlocked set?

Maverick10/17/25, 5:16 AM

Crawlee generates the following warning when we set both blockedStatusCodes and retryOnBlocked
'Both blockedStatusCodes and retryOnBlocked are set. Please note that the retryOnBlocked feature might not work as expected'

When I set retryOnBlocked to true, crawlee automatically sets default blocked status codes for the session pool an empty array.
When I do handleCloudflareChallenge inside Post navigation hooks, I get 403 status code. Even though handleCloudflareChallenge tries to remove 403 from sessionpool's blockedStatusCodes since it is empty it doesn't have any effect.

Since retryOnBlocked is true, crawlee checks the status code and if its length is zero, it compares with default status codes to check whether to throw an error. And it always throws error.

Is this expected behaviour? If so can I override default blockedStatusCodes? Is there a problem with it?

Upwork Job Opportunity

╲

╲⎝⧹ W҉O҉L҉F҉ ⧸⎠╱10/16/25, 5:18 PM

If you are US citizen and share your Upwork. You can finish $500 budget - contract immediately with my help. My client and I am ready!
I hope you collaborate with me.
Please DM me.

Use Browser Rendering from cloudflare workers

Eduardo10/15/25, 11:40 AM

Is posible to use cf workers with https://developers.cloudflare.com/browser-rendering/get-started/ ?

Cloudflare Docs

Get started

Cloudflare Browser Rendering allows you to programmatically control a headless browser, enabling you to do things like take screenshots, generate PDFs, and perform automated browser tasks.

Best practices for long living crawler & rabbitmq

Cypher10/11/25, 4:42 PM

Hi guys,
I’m here to ask best practices. I have coded a simple manager:

starts crawlee instance and initializes queue object
receives rabbitmq messages and pushes them to the queue

This is set up as keep alive crawler (it never quits, just awaits new messages). Thing is that it has some cache and some work around the queue and crawler to not actually let it close (for some reason it was just stopping and new messages were pushed tu queue but it wasn’t read by the crawler)

This made me wonder, maybe it should be built different?

Is there any resource that would help me learn about best practices in building such thing on crawlee? Docs lack long living crawler examples

I’ll add that my setup is using many different handlers for different sites - don’t know if it’s important for this question

Camoufox can't establish connection with a websocket server

Nobre9/28/25, 4:13 PM

addRequestsBatched isn’t working for me on Apify, even though it works perfectly fine locally

azzouzana9/28/25, 1:41 PM

Hi, I’m having trouble with

addRequestsBatched

addRequestsBatched

(& also

addRequests

addRequests

) It works perfectly fine when I run it locally, but it on Apify it does not seem to do anything. Has anyone faced this issue or knows what might be causing it?
I'm using the exact same versions of node (22.19.0) , crawlee (3.15.1) & apify (3.4.5)

Key-Value-Store Issue

Umit9/25/25, 9:57 PM

Does anyone know how to use

writeStream

writeStream

to save files into key-value-store?
It seems that it does work, however file do not show up when running on the Apify platform?

Also, saving the files usually with

await <kvs>.setValue('<file-key-here>,<Buffer>,<content-type>)

await <kvs>.setValue('<file-key-here>,<Buffer>,<content-type>)

;
Errors saying that the data

JSON.stringify()

JSON.stringify()

returned undefined

This is despite the data being checked before it was passed on like so:

let data = JSON.stringify(body)

if(data){
await setValue(<key>, body , content-type.type
}

let data = JSON.stringify(body)

if(data){
await setValue(<key>, body , content-type.type
}

Any suggestions?

Customize Run Options(memory and time out) in Playwright Node.js Crawler Actor

Exp9/25/25, 3:04 PM

Hello everyone
I’ve been running into some issues when trying to customize the Run Options for my Apify Actor, especially with increasing the allocated memory.
Does anyone know if it’s possible to configure or increase the memory amount directly from within a PlaywrightCrawler (Crawlee) Node.js code? Or is this something that can only be set through the Apify Console / API run options?
If you’ve had experience dealing with this, I’d really appreciate any insights or best practices you can share

Limit request queue

NeoNomade9/18/25, 11:58 AM

I have some crawlers that are consuming from RabbitMQ, but they obviously take all the messages from Rabbit and move them to the internal queue. Can I somehow cap the requestQueue ? so it can only take a finite amount of requests ?

import { Router } from "crawlee"; export const router = Router.create(); // ... other handlers // imagine the category handler is called twice router.addHandler("CATEGORY", async ({ enqueueLinks }) => { // ... parsing // result of first call await enqueueLinks({ urls: ["https://example.com/my-product"], userData: { category: "foo" }, label: "DETAIL", }); // result of second call await enqueueLinks({ urls: ["https://example.com/my-product"], userData: { category: "bar" }, label: "DETAIL", }); }); router.addHandler("DETAIL", async ({ request }) => { // todo: how to get both "foo" and "bar" here? const category = request.userData.category; // ... saving });