NeoNomadeN
Apify & Crawlee•3y ago•
11 replies
NeoNomade

isTaskReadyFunction failing randomly

I've built a Cheerio Crawler that doesn't do anything super fancy, it takes a start url, than it has 2 enqueue links functions, and another handler that saves to the dataset the url and the body of the page.
I've exposed the GC and running it after both of the request handlers, and also where I'm saving the body , I'm assigning the body to null after saving it.

But I get this error randomly, sometimes at the beginning of the script, sometimes after 20k items scraped sometimes after 50k items scraped, but I could never pass the 50-55k items.
MacOS Ventura 13.1
Node v19.6.0 || npm 9.4.1


ERROR CheerioCrawler:AutoscaledPool: isTaskReadyFunction failed
  SyntaxError: Unexpected end of JSON input
      at JSON.parse (<anonymous>)
      at RequestQueueFileSystemEntry.get (/Users/user/project/node_modules/@crawlee/memory-storage/fs/request-queue/fs.js:19:21)
      at async RequestQueueClient.listHead (/Users/user/project/node_modules/@crawlee/memory-storage/resource-clients/request-queue.js:147:29)
      at async RequestQueue._ensureHeadIsNonEmpty (/Users/user/project/node_modules/@crawlee/core/storages/request_queue.js:610:101)
      at async RequestQueue.isEmpty (/Users/user/project/node_modules/@crawlee/core/storages/request_queue.js:526:9)
      at async CheerioCrawler._isTaskReadyFunction (/Users/user/project/node_modules/@crawlee/basic/internals/basic-crawler.js:710:38)
      at async AutoscaledPool._maybeRunTask (/Users/user/project/node_modules/@crawlee/core/autoscaling/autoscaled_pool.js:481:27)
Was this page helpful?