relationship between memoryMbytes and availableMemoryRatio?

Can someone tell me what the relationship between those two configuration options is? is one jut in MB and the other in a ratio of the total?

I thought availableMemoryRatio limited the total memory used, but "When the memory usage is more than the provided ratio, the memory is considered overloaded." so it's more like an indicator? I'm currently working on a heavily overloaded server, and would rather that crawlee/playwright had a strong upper limit and was slower.

faint-whiteOP•10/31/23, 10:05 AM

and with overloaded I mean it's using 600% of the available memory

Lukas Krivka•11/1/23, 10:41 AM

is not a memory limit but autoscaling config telling it when you scale up or down. What you probably want to set set the max memory for the whole process.

faint-whiteOP•11/1/23, 2:37 PM

yes, is there a way to do that?

Lukas Krivka•11/1/23, 8:09 PM

Try this - https://crawlee.dev/docs/guides/configuration#crawlee_memory_mbytes

Configuration | Crawlee

Configuring Crawlee parameters

Lukas Krivka•11/1/23, 8:09 PM

You can set it as env var or in the code via Configuration class

faint-whiteOP•11/2/23, 8:54 AM

I have not set that, but the docs mention this:

By default, the max amount of memory to be used is set to one quarter of total system memory, i.e. on a system with 8192 MB of memory, the autoscaling feature will only use up to 2048 MB of memory.

yet it's going well beyond 25% of the available memory, so it doesn't look like it limits memory usage

MEE6•11/2/23, 8:54 AM

@Kilian just advanced to level 3! Thanks for your contributions!

Lukas Krivka•11/2/23, 10:59 AM

Its does not really put a hard limit, it will just not scale up in that case. If you are doing a lot of work inside a handler or there is sudden spike, it can go up.

faint-whiteOP•11/2/23, 2:14 PM

so there's no way to hard-cap it?

Lukas Krivka•11/2/23, 2:46 PM

You can do that on Node.js level, see https://support.circleci.com/hc/en-us/articles/360009208393-How-Can-I-Increase-the-Max-Memory-for-Node-

CircleCI Support Center

How Can I Increase the Max Memory for Node?

Default Memory Size
Node has a default max memory usage of less than 2GB on some 64bit systems (depending on the Node runtime version). This can cause unexpected memory issues when running on Circl...