Ddual-salmon
autoscale pool trying to scale up without suffecient memory
Hi All,
im running a playwright crawler and am running into a bit of an issue with crawler stability. Have a look at these two log messages
autoscaled pool is trying to increase its concurrency from 4 to 5 since it was in its view idle. 20 seconds later though
which suggests memory was much tighter than autoscaledpool was considering, likley due to the additional ram that chromium was using. Crawlee was running in a k8 pod with a 4GB ram limit. Is this behaviour intended and how might i improve my performance? Does autoscaled pool account for how much ram is actually in use or just how much the node process uses?
im running a playwright crawler and am running into a bit of an issue with crawler stability. Have a look at these two log messages
autoscaled pool is trying to increase its concurrency from 4 to 5 since it was in its view idle. 20 seconds later though
which suggests memory was much tighter than autoscaledpool was considering, likley due to the additional ram that chromium was using. Crawlee was running in a k8 pod with a 4GB ram limit. Is this behaviour intended and how might i improve my performance? Does autoscaled pool account for how much ram is actually in use or just how much the node process uses?
