Vverbal-lime
Why the CPU utilization rate of crawlee is going down and seem like stop processing any requests
This always happens after one hour running. Here is the link for most of the code https://medium.com/p/how-to-efficiently-scrape-millions-of-google-businesses-on-a-large-scale-using-a-distributed-35b9140030eb
Most of them are c5d.2xlarge SPOT instances. In order to solve this issue, I have to build a cron task to find out the instances with low cpu utilization and terminate them.
Most of them are c5d.2xlarge SPOT instances. In order to solve this issue, I have to build a cron task to find out the instances with low cpu utilization and terminate them.
Medium
