Newer Crawlee Versions Cause Double Request Queue Writes & Version Compatibility Issues
🍩CheerioCrawler
I'm experiencing a critical cost issue where Crawlee 3.15.3 writes 2x more request queue operations compared to version 3.12.1 for the exact same actor and input on Apify platform. This doubles my costs, so I need to use 3.12.1, but I'm hitting compatibility issues.
When trying to use
1. TypeScript compilation error - Type mismatch in proxy configuration:
2. Runtime error when running locally (even after ignoring TypeScript with
Cost Impact: For the same run with identical input, Crawlee 3.15.3 generates roughly double the request queue writes compared to 3.12.1:
(See attached screenshot)
I don't know what changed between these versions, but 3.15.3 is significantly more expensive to run.
To force deduplication locally and replicate Apify's behavior, I added npm overrides to package.json:
1. Is using npm "overrides" the correct approach here? Will this cause issues when deploying to Apify platform?
2. Why does Crawlee 3.15.3 generate 2x more request queue writes than 3.12.1? What changed between these versions that causes this cost increase?
The Problem (When running locally on my machine)
When trying to use
crawlee@3.12.1crawlee@3.12.1 with apify@^3.0.0apify@^3.0.0 (3.2.6), I encounter two blocking issues1. TypeScript compilation error - Type mismatch in proxy configuration:
2. Runtime error when running locally (even after ignoring TypeScript with
as anyas any):Why I Need 3.12.1 Specifically
Cost Impact: For the same run with identical input, Crawlee 3.15.3 generates roughly double the request queue writes compared to 3.12.1:
(See attached screenshot)
I don't know what changed between these versions, but 3.15.3 is significantly more expensive to run.
My Current Solution (Need Validation)
To force deduplication locally and replicate Apify's behavior, I added npm overrides to package.json:
My Questions
1. Is using npm "overrides" the correct approach here? Will this cause issues when deploying to Apify platform?
2. Why does Crawlee 3.15.3 generate 2x more request queue writes than 3.12.1? What changed between these versions that causes this cost increase?
