3 replies

Why the CPU utilization rate of crawlee is going down and seem like stop processing any requests

This always happens after one hour running. Here is the link for most of the code https://medium.com/p/how-to-efficiently-scrape-millions-of-google-businesses-on-a-large-scale-using-a-distributed-35b9140030eb
Most of them are c5d.2xlarge SPOT instances. In order to solve this issue, I have to build a cron task to find out the instances with low cpu utilization and terminate them.

CRAWLEE_MIN_CONCURRENCY: "3"
CRAWLEE_MAX_CONCURRENCY: "15"
CRAWLEE_MEMORY_MBYTES: "4096"

CRAWLEE_MIN_CONCURRENCY: "3"
CRAWLEE_MAX_CONCURRENCY: "15"
CRAWLEE_MEMORY_MBYTES: "4096"

Medium

Support me on (Patreon)[https://www.patreon.com/tonywang_dev] to write more tutorials like this!