Apify Discord Mirror

Updated last month

Hanging Chromium processes with Playwright in multi-worker setup

Wojciech · 2024-10-24T09:18:25.506Z

Environment:Node.js multi-threaded application with worker poolRunning on AWS (4GB RAM)Large-scale web crawling (10,000+ websites)Processing in batches of 10-20 URLsCurrent Behavior:Application crashes after ~15-20k websites (memory overload)~180 zombie Chromium processes remain after teardown()Can't safely kill processes due to shared worker environmentProcess Flow:Fetch 1000 URLs from DBCreate batches (10-20 URLs each) ( divided by userId for easier maintainance )Process each batch:// Pseudo flow addBatchToQueue() runCrawler() await teardown()Key Challenges:teardown() isn't effectively cleaning up Chromium instancesCan't manually kill processes (would affect other workers)No native way to track specific Playwright instance PIDsQuestions:Has anyone encountered similar memory leaks with heavy crawler loads?Are there recommended patterns for proper cleanup in multi-worker scenarios?Any suggestions for tracking/managing Chromium processes per Playwright instance?System Info:Load average: 0.11 0.13 0.09Memory usage: ~3.25GB/3.74GBScreenshots of htop attached showing the hanging processes. Any insights would be greatly appreciated! 🙏

At a glance

WWojciech

·

This post has been deleted.

Add a reply

Sign up and join the conversation on Discord