Memory usage is spiking, even with CRAWLEE_AVAILABLE_MEMORY_RATIO set to .7
Memory usage is spiking, even with CRAWLEE_AVAILABLE_MEMORY_RATIO set to .7
At a glance
The post asks for ideas on how to solve an issue, and the community members provide the following suggestions:
1. Provide logs, links to the run, or examples of the mentioned issue to get more insight.
2. Limit the maxConcurrency or the supplied memory, and slow down the scaling up.
3. Ensure the container is configured with the correct memory limit (8GB) and that the available_memory_ratio is handling it properly.
4. The concurrency should slow down as it reaches CPU/memory/event loop utilization, but spikes can still happen. Examine the logs to see if concurrency is rapidly increasing before the spike.
There is no explicitly marked answer in the comments.
That controls the concurrency so it should slow down while reaching CPU/memory/event loop utilization. But spikes can still happen if you have bad luck.
First look at the logs, is concurrency rapidly increasing before you see the spike? We need more insight into what those spikes are
