ugly-tan•2y ago

Is there way to not store crawled data in the crawler?

Getting this warning in production may be because of storage - WARN PlaywrightCrawler:AutoscaledPool:Snapshotter: Memory is critically overloaded. Using 2652 MB of 512 MB (518%). Consider increasing available memory. How can I remove this?

3 Replies

HonzaS•2y ago

It is not memory for crawled data but memory used for the run.

ugly-tanOP•2y ago

Ah okay. Can it be optimised in some way?

HonzaS•2y ago

lower the concurrency

Is there way to not store crawled data in the crawler?

Did you find this page helpful?