Ssacred-emerald
Large threaded, kubernetes scrape = Target page, context or browser has been closed
Ironically @cryptorex just posted a similar issue (TargetClosedError: Target page, context or browser has been closed (I've tried a lot)) but I wanted to provide some additional context to see if they're related.
I'm:
- Running a node app with worker threads (usually 32 of them)
- Running multiple containers in kubernetes
Each thread:
- Grabs 5 domains from my postgres DB (of 5 million!)
- Loops through each domain
- Creates a new PlaywrightCrawler with unique-named storages (to prevent collision / global deletion from crawlers in other threads)
- Queues the domains home page
- Controllers then queue up some additional pages based on what's found on the home page
- The results are processed in real-time and pushed to the database (since we don't want to wait until all 5M all are complete
- The thread-specific sotrages are then deleted used drop()
The Problem
This works flawlessly... for about 60 minutes... afterwards, I get plagued with
What I've tried:
-
-
- A cron to restart my cluster
- Ensuring the EBS volumes are not running out of space (they're 20GB each and seem to be 50% full when crashing)
- Ensuring the pods have plenty of memory (running EC2s with 64GB memory and 16 CPU (32 threads). Seems to handle the load in the first hour just fine.
I suspect there's a leak or store not being cleared out since it happens gradually?
I'm:
- Running a node app with worker threads (usually 32 of them)
- Running multiple containers in kubernetes
Each thread:
- Grabs 5 domains from my postgres DB (of 5 million!)
- Loops through each domain
- Creates a new PlaywrightCrawler with unique-named storages (to prevent collision / global deletion from crawlers in other threads)
- Queues the domains home page
- Controllers then queue up some additional pages based on what's found on the home page
- The results are processed in real-time and pushed to the database (since we don't want to wait until all 5M all are complete
- The thread-specific sotrages are then deleted used drop()
The Problem
This works flawlessly... for about 60 minutes... afterwards, I get plagued with
Target page, context or browser has been closedWhat I've tried:
-
browserPoolOptionsretireBrowserAfterPageCount: 100closeInactiveBrowserAfterSecs: 200-
await crawler.teardown();- A cron to restart my cluster
- Ensuring the EBS volumes are not running out of space (they're 20GB each and seem to be 50% full when crashing)
- Ensuring the pods have plenty of memory (running EC2s with 64GB memory and 16 CPU (32 threads). Seems to handle the load in the first hour just fine.
I suspect there's a leak or store not being cleared out since it happens gradually?
This is the official developer community of Apify and Crawlee.
14,091Members
Resources
Recent Announcements
Similar Threads
Was this page helpful?
