EAdaptativePlaywright crawler with SQL queue gets stuck
I am using multiple instances of an AdaptativePlaywright crawler in different servers, using the SQL queue. After running for some time, it looks like the crawler stops working and I only regularly see this logs:
│ [crawlee._autoscaling.autoscaled_pool] WARN Task timed out after not set seconds │
│ [crawlee._autoscaling.autoscaled_pool] INFO current_concurrency = 9; desired_concurrency = 10; cpu = 0.0; mem = 0.0; event_loop = 0.023; client_info = 0.0 │
│ [crawlee._autoscaling.autoscaled_pool] WARN Task timed out after not set seconds │
│ [crawlee.crawlers._basic._basic_crawler] INFO Crawled 0/1157424 pages, 0 failed requests, desired concurrency 10.
Restarting the pod where the crawler runs doesn't fix the issue
│ [crawlee._autoscaling.autoscaled_pool] WARN Task timed out after not set seconds │
│ [crawlee._autoscaling.autoscaled_pool] INFO current_concurrency = 9; desired_concurrency = 10; cpu = 0.0; mem = 0.0; event_loop = 0.023; client_info = 0.0 │
│ [crawlee._autoscaling.autoscaled_pool] WARN Task timed out after not set seconds │
│ [crawlee.crawlers._basic._basic_crawler] INFO Crawled 0/1157424 pages, 0 failed requests, desired concurrency 10.
Restarting the pod where the crawler runs doesn't fix the issue
This is the official developer community of Apify and Crawlee.
14,091Members
Resources
Recent Announcements
Similar Threads
Was this page helpful?
