Apify Discord Mirror

Updated 2 years ago

How to increase max memory?

At a glance

The community member is running a script that requires concurrency and has 64 GB of RAM available, but is encountering a memory overloaded error at around 15 GB. They have tried setting the memoryMbytes and availableMemoryRatio configuration options, but the behavior has not changed. Another community member suggests trying to run the script with the CRAWLEE_MEMORY_MBYTES environmental variable set, based on the information in the provided link.

Useful resources

JJeno

I am running a script that needs concurrency. I have 64 GB of RAM available and I want to use it to the max. I am running my script on a server so there is not much else running. The problem is, at around 15GB I always get memory overloaded error.

I have tried:

Plain Text

config.set('memoryMbytes', 50_000)
config.set('availableMemoryRatio', 0.95)

Nothing seems to change this behavior. Anything else I can try?

1 comment

PPepa J

Hello Jeno, based on https://sdk.apify.com/api/apify/interface/ConfigurationOptions#memoryMbytes it seems to me, that in your case, there is still the default memory limit being used.

Have you tried to run the script with CRAWLEE_MEMORY_MBYTES environmental variable being set?

Add a reply