Memory for only 1 browser is 12gb? How to ensure clean up after pages?
Memory for only 1 browser is 12gb? How to ensure clean up after pages?
At a glance
The post asks whether the user needs to manually call page.close() or anything at the end of the default handler. The community members discuss that the pages are closed automatically once the handler function is done executing. However, some community members mention that there can be a significant difference in RAM usage and speed when closing pages manually versus letting them be closed automatically, especially for larger-scale crawlers (5-10k pages).
The community members also discuss issues with Crawlee's auto-scaling and memory management, where some users report getting "memInfo":{"isOverloaded":true,"limitRatio":0.2,"actualRatio":1} even without running anything, and the Snapshotter component always leaking memory. They suggest that the issues may be related to the operating system, as MacOS and Alpine Linux seem to handle the scaling better than other OSes. The community members request a reproducible example to investigate the issue further.
probably there is a designed process to do that. But on a bigger scale, is a big difference on ram usage when closing page manually and auto handling this. For spiders between 5-10k pages where I tested, the difference in ram memory and speed is huge
The memory usage should scale with concurrency. If the memory is increasing but the concurrency is decreasing that can mean memory leak either in your code or in Crawlee. We need a reproduction in that case. Generally, you should be able to run crawler for days without problems
this is completely not what's happening -- i'm getting "memInfo":{"isOverloaded":true,"limitRatio":0.2,"actualRatio":1} even without running anything. Snapshotter is always leaking. I don't know how to reproduce.
I'm on holidays, I'll get back to my laptop later tonight. I have hundreds of Crawlee spiders running daily on Alpine (mostly Puppeteer and Cheerio) that will try to scale infinitely. If I don't limit max concurrency, AWS batch is killing the jobs automatically . I think the easiest way to reproduce, is to take the demo spider for the crawlee page and just replace the Dockerfile with an Alpine one. Then run the container with limited resources something like 2vcpu and 4gb of ram. It will be very easy to see that autoscailing fails
By default Crawlee is saying that if I not mention maxConcurrency it will scale up to available resources and I'm telling you that on many OSs it is not able to read the resources. I have containers killed with OOM if I don't use maxConcurrency based on previous benchmarking that I've done myself.
On Linux max requests per minute is never working. It's outputting a bunch of logs and it gets stuck (I'll provide an example in approx 8-10 hours)
Also not related to this, on Linux for the moment neither the request locking works properly.
Thank you for your explanation. If you would be able to provide minimal PoC repository, where this problem occur and we may investigate this, that would help a lot.
