xenial-black•2y ago
Memory usage is spiking, even with CRAWLEE_AVAILABLE_MEMORY_RATIO set to .7
Any ideas on how to solve this?
4 Replies
Can You please provide any logs / links to your run / examples of mentioned issue?
Limit
maxConcurrency or the supplied memory. You can also slow down the scaling up but it is kinda trying stuff outxenial-blackOP•2y ago
I have the container giving 8gb, do I have to tell it that it only has 8bb? I thought available_memory_ratio would handle.
That controls the concurrency so it should slow down while reaching CPU/memory/event loop utilization. But spikes can still happen if you have bad luck.
First look at the logs, is concurrency rapidly increasing before you see the spike? We need more insight into what those spikes are