fierDeToiMonGrandF
Apify & Crawlee•2w ago•
3 replies
fierDeToiMonGrand

Newer Crawlee Versions Cause Double Request Queue Writes & Version Compatibility Issues

I'm experiencing a critical cost issue where Crawlee 3.15.3 writes 2x more request queue operations compared to version 3.12.1 for the exact same actor and input on Apify platform. This doubles my costs, so I need to use 3.12.1, but I'm hitting compatibility issues.

The Problem (When running locally on my machine)


When trying to use 
crawlee@3.12.1
 with 
apify@^3.0.0
 (3.2.6), I encounter two blocking issues

1. TypeScript compilation error - Type mismatch in proxy configuration:

const proxyConfiguration = await Actor.createProxyConfiguration();
const crawler = new CheerioCrawler({
  proxyConfiguration: proxyConfiguration, // ❌ Type error
  requestHandler: router,
});

2. Runtime error when running locally (even after ignoring TypeScript with 
as any
):

Error: Detected incompatible Crawlee version used by the SDK.
User installed 3.12.1 but the SDK uses 3.15.3.
    at checkCrawleeVersion (node_modules/apify/utils.js:68)
    at Actor.init (node_modules/apify/main.js:123)


Why I Need 3.12.1 Specifically


Cost Impact: For the same run with identical input, Crawlee 3.15.3 generates roughly double the request queue writes compared to 3.12.1:

(See attached screenshot)

I don't know what changed between these versions, but 3.15.3 is significantly more expensive to run.

My Current Solution (Need Validation)


To force deduplication locally and replicate Apify's behavior, I added npm overrides to package.json:

{
  "dependencies": {
    "apify": "^3.0.0",
    "crawlee": "3.12.1"
  },
  "overrides": {
    "@crawlee/core": "3.12.1",
    "@crawlee/types": "3.12.1",
    "@crawlee/utils": "3.12.1"
  }
}


My Questions


1. Is using npm "overrides" the correct approach here? Will this cause issues when deploying to Apify platform?
2. Why does Crawlee 3.15.3 generate 2x more request queue writes than 3.12.1? What changed between these versions that causes this cost increase?
image.png
image.png
Was this page helpful?