Apify & Crawlee

This is the official developer community of Apify and Crawlee.

Join

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

Join

crawlee-js

apify-platform

crawlee-python

💻hire-freelancers

🚀actor-promotion

💫feature-request

💻creators-and-apify

🗣general-chat

🎁giveaways

programming-memes

🌐apify-announcements

🕷crawlee-announcements

👥community

azzouzana

10/25/2025

Impit client: how to set http2, and how can I order headers & pseudo headers chirurgically

Basically the title

Casper

10/22/2025

Problem with scraping a site that requires login

I have a paid actor I am renting out to customers that is failing because of a recent anti bot mitigation that prevents scraping pages past 10 without logging in. I have implemented Google login and store session cookies in a shared key value store for the actor to use and this seem to work fine. However Google has flagged account and logins as being a bot and has since terminated the account thus login fails and then scraping fails as well. Before the Google account termination, I experienced that the site I scrape, also seemed to throttle my requests - however this is without using a proxy so might be possible to circumvent, however this has never been an issue before with this site. The site has option for Google, Facebook, Apple or email login and I chose Google because email requires to receive a login code to the email each time a login is performed, which I couldn't automate. I have been trying to resolve this for the past week and was successfull until the Google login termination....

Louis Deconinck

10/21/2025

How would you log a nicely formatted full JSON response for debugging?

How would you log a nicely formatted full JSON response for debugging using the log?

Louis Deconinck

10/21/2025

How to use Impit as the http client for Crawlee?

How can I use Impit as the http client when using the httpcrawler?

Maverick

10/17/2025

Why does crawlee generate warning when both blockedStatusCodes and retryOnBlocked set?

Crawlee generates the following warning when we set both blockedStatusCodes and retryOnBlocked 'Both blockedStatusCodes and retryOnBlocked are set. Please note that the retryOnBlocked feature might not work as expected' When I set retryOnBlocked to true, crawlee automatically sets default blocked status codes for the session pool an empty array. When I do handleCloudflareChallenge inside Post navigation hooks, I get 403 status code. Even though handleCloudflareChallenge tries to remove 403 from sessionpool's blockedStatusCodes since it is empty it doesn't have any effect....

╲⎝⧹ W҉O҉L҉F҉ ⧸⎠╱

10/16/2025

Upwork Job Opportunity

If you are US citizen and share your Upwork. You can finish $500 budget - contract immediately with my help. My client and I am ready! I hope you collaborate with me. Please DM me....

Eduardo

10/15/2025

Use Browser Rendering from cloudflare workers

Is posible to use cf workers with https://developers.cloudflare.com/browser-rendering/get-started/ ?

Cypher

10/11/2025

Best practices for long living crawler & rabbitmq

Hi guys, I’m here to ask best practices. I have coded a simple manager: - starts crawlee instance and initializes queue object - receives rabbitmq messages and pushes them to the queue ...

Nobre

9/28/2025

Camoufox can't establish connection with a websocket server

azzouzana

9/28/2025

addRequestsBatched isn’t working for me on Apify, even though it works perfectly fine locally

Hi, I’m having trouble with addRequestsBatched (& also addRequests) It works perfectly fine when I run it locally, but it on Apify it does not seem to do anything. Has anyone faced this issue or knows what might be causing it? I'm using the exact same versions of node (22.19.0) , crawlee (3.15.1) & apify (3.4.5)...

Umit

9/25/2025

Key-Value-Store Issue

Does anyone know how to use writeStream to save files into key-value-store? It seems that it does work, however file do not show up when running on the Apify platform? Also, saving the files usually with await <kvs>.setValue('<file-key-here>,<Buffer>,<content-type>) ; Errors saying that the data JSON.stringify() returned undefined...

Exp

9/25/2025

Customize Run Options(memory and time out) in Playwright Node.js Crawler Actor

Hello everyone I’ve been running into some issues when trying to customize the Run Options for my Apify Actor, especially with increasing the allocated memory. Does anyone know if it’s possible to configure or increase the memory amount directly from within a PlaywrightCrawler (Crawlee) Node.js code? Or is this something that can only be set through the Apify Console / API run options? If you’ve had experience dealing with this, I’d really appreciate any insights or best practices you can share...

NeoNomade

9/18/2025

Limit request queue

I have some crawlers that are consuming from RabbitMQ, but they obviously take all the messages from Rabbit and move them to the internal queue. Can I somehow cap the requestQueue ? so it can only take a finite amount of requests ?

NeoNomade

9/17/2025

RabbitMQ and RequestQueue or List

Did anybody managed to actually straight connect the RequeustQueue or RequestList to a continuous consumption process from a RabbitMQ queue ?

9/13/2025

Initializing CloudFlare cookies with Crawlee

Hi, I am currently using a PlaywrightScrapper to initialize cloudflare cookies to then send requests to the website programmatically. My problem is that the target website does multiple redirects to itself before the CF cookie is ready, which I did not achieve to handle using my code. You know the cookie is ready when you get a 200 status code....

strock77

9/8/2025

Safe to parallelize Dataset writes across processes?

Context: • Crawlee v3.13.10, Node 22 • Linux (ext4), using storage-local • Multiple forked workers share one RequestQueueV2 (with request locking) • Each worker does:...

aydin

9/6/2025

Crawlee Hybrid Crawler?

I notice a lot of the time I end up writing the exact same type of crawler where it first uses CheerioCrawler and then falls back to PlaywrightCrawler for failed requests. The only annoying thing is the obviously different syntax between cheerio and playwright ($ and load for Cheerio vs page for Playwright). For code reuse purposes i end up writing a lot of code that looks like this

...(crawlerType === 'playwright' ? { launchContext: getLaunchContext() } : {}),

...(crawlerType === 'playwright' ? { launchContext: getLaunchContext() } : {}),

...

Jack Wenyoung

9/6/2025

How to combine the scraping results for Crawlee Playwright actor?

Hello folks! I'm building an actor with Crawlee and playwright for bbb scraping. I have completed the "detail" content parsing, and I made two booleans inputs to let users decide whether to scrape reviews, complaints or not. Then I realized if they choose both, I'll need to send two more requests and I don't know how to use the router to combine the results(the reviews, complaints and detail results) before using Dataset.pushData....

Solution:

You need a way to persist state per entity (e.g., one business from BBB) across multiple requests, and only pushData() once all requested pieces (detail + reviews + complaints) are collected. In Crawlee, you do this by: ``` 1. Storing a “partial result” in request.userData....

Prime

9/6/2025

Crawl sitemap

Hi guys i am trying to extract links from sitemap it works fine when i call for first time but if i call it again with same sitemap url it not extract links from sitemap again what is the problem ? this is my code ```js async crawlSitemap({ url,...

Solution:

nvm i add

const config = new Configuration({ persistStorage: false });

const config = new Configuration({ persistStorage: false });

and pass it to PlaywrightCrawler fix...

Martin

9/2/2025

FAcebook Ads library src video/images

Hi everyone, I'm building a scraper using Apify for the Facebook Ads Library. I'm fetching ad data via the Ads Library API, which provides details including an ad_snapshot_url. The issue is that the direct URL for the ad creative (the image or video file) is not included in the API response. My approach is to open the ad_snapshot_url with Playwright and attempt to extract the <img> or <video> element from the DOM....