Memory is critically overloaded. Using 12184 MB of 3883 MB (314%). Consider increasing available mem
Memory is critically overloaded. Using 12184 MB of 3883 MB (314%). Consider increasing available mem
At a glance
The community member is running a Playwright crawler on a Linux VPS with 8 cores and 15533MB of memory, but is receiving warnings about critical memory overload, using 12184MB of 3883MB (314%). The community members have suggested configuring the CRAWLEE_MEMORY_MBYTES environment variable to a higher value and using the --max-old-space-size attribute to run the spider. They have also discussed issues with not closing pages, mixing scraping and PDF parsing, and the need to divide processes and clean up the code. The community members have offered to provide further assistance by reviewing the code and having a call to troubleshoot the issue.
Hi. I am running a playwright crawler in my linux vps. The vps ihas 8 core CPU and 15533MB memory. But I got many warning like : WARN PlaywrightCrawler:AutoscaledPool:Snapshotter: Memory is critically overloaded. Using 12184 MB of 3883 MB (314%). Consider increasing available memory.
Don't know what package you are using for that. Check docs of that particular package. Also I don't think those activities should be mixed. Scraping is scraping, pdf parsing is something different . Indeed memory usage can go bananas if you scrape and read pdfs and mix in some other stuff like maybe unclosed connections to S3. Divide your processes, clean the code .
Totally agree, we don't read the PDF's on the same machine. Sorry that was convoluted. We hit a lot of PDF's and then either intercept request and send the url to SQS to download and parse.
