Memory is critically overloaded

I have an AWS EC2 instance with 64GB memory.
My crawler is running in a docker container. The CRAWLEE_MEMORY_MBYTESCRAWLEE_MEMORY_MBYTES is set to 6144061440

My docker config

docker run --rm --init -t $docker_args \
    -v /mnt/storage:/app/storage \
    --user appuser \
    --security-opt seccomp=/var/lib/jenkins/helpers/docker/seccomp_profile.json \
    -e MONGO_HOST=${MONGO_HOST} \
    -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    -e SPAWN=${SPAWN} \
    -e MONGO_CACHE=${MONGO_CACHE} \
    -e CRAWLEE_MEMORY_MBYTES=${CRAWLEE_MEMORY_MBYTES} \
    ${IMAGE_NAME} $prog_args

docker run --rm --init -t $docker_args \
    -v /mnt/storage:/app/storage \
    --user appuser \
    --security-opt seccomp=/var/lib/jenkins/helpers/docker/seccomp_profile.json \
    -e MONGO_HOST=${MONGO_HOST} \
    -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    -e SPAWN=${SPAWN} \
    -e MONGO_CACHE=${MONGO_CACHE} \
    -e CRAWLEE_MEMORY_MBYTES=${CRAWLEE_MEMORY_MBYTES} \
    ${IMAGE_NAME} $prog_args

After crawling around 15-20 minutes I receive the warning Memory is critically overloadedMemory is critically overloaded. But when doing the command free -hfree -h I notice I still have a lot of free memory as you can see in the screenshots.

The playwright crawler is configured like this

 crawler = PlaywrightCrawler(
        concurrency_settings=ConcurrencySettings(
            min_concurrency=25,
            max_concurrency=125,
        ),
        request_manager=request_queue,
        request_handler=router,
        retry_on_blocked=True,
        headless=True,
    )

 crawler = PlaywrightCrawler(
        concurrency_settings=ConcurrencySettings(
            min_concurrency=25,
            max_concurrency=125,
        ),
        request_manager=request_queue,
        request_handler=router,
        retry_on_blocked=True,
        headless=True,
    )

I'm a bit clueless why I have this issue. In the past I had issues with zombie processes but this was solved by adding the --init--init to my docker run command.
Do you have any idea on how I can fix this or further debug this?

Apify & Crawlee•13mo ago•

4 replies

recent-teal

Memory is critically overloaded

I have an AWS EC2 instance with 64GB memory.
My crawler is running in a docker container. The CRAWLEE_MEMORY_MBYTESCRAWLEE_MEMORY_MBYTES is set to 6144061440

My docker config

docker run --rm --init -t $docker_args \
    -v /mnt/storage:/app/storage \
    --user appuser \
    --security-opt seccomp=/var/lib/jenkins/helpers/docker/seccomp_profile.json \
    -e MONGO_HOST=${MONGO_HOST} \
    -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    -e SPAWN=${SPAWN} \
    -e MONGO_CACHE=${MONGO_CACHE} \
    -e CRAWLEE_MEMORY_MBYTES=${CRAWLEE_MEMORY_MBYTES} \
    ${IMAGE_NAME} $prog_args

docker run --rm --init -t $docker_args \
    -v /mnt/storage:/app/storage \
    --user appuser \
    --security-opt seccomp=/var/lib/jenkins/helpers/docker/seccomp_profile.json \
    -e MONGO_HOST=${MONGO_HOST} \
    -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    -e SPAWN=${SPAWN} \
    -e MONGO_CACHE=${MONGO_CACHE} \
    -e CRAWLEE_MEMORY_MBYTES=${CRAWLEE_MEMORY_MBYTES} \
    ${IMAGE_NAME} $prog_args

 crawler = PlaywrightCrawler(
        concurrency_settings=ConcurrencySettings(
            min_concurrency=25,
            max_concurrency=125,
        ),
        request_manager=request_queue,
        request_handler=router,
        retry_on_blocked=True,
        headless=True,
    )

 crawler = PlaywrightCrawler(
        concurrency_settings=ConcurrencySettings(
            min_concurrency=25,
            max_concurrency=125,
        ),
        request_manager=request_queue,
        request_handler=router,
        retry_on_blocked=True,
        headless=True,
    )

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,851 Members

Join

Memory is critically overloaded

Memory is critically overloaded

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads