Node-cron with CheerioCrawler
I'm currently using CheerioCrawler along with node-cron to run my web scraping tasks. However, I've been having an issue with the crawler not stopping once it's done with its tasks.
I set up a node-cron job for every 30 seconds, but the problem occurs when the crawler stays open after finishing its tasks. When the cron job arrives, it seems to creates another instance, and the first iteration works fine. However, subsequent iterations do not start scraping all the pages as expected.
In the terminal, it says that the crawler has finished its tasks, but it does not start scraping again. I know that I will eventually need to run it for a longer duration, such as 30 minutes. Therefore, I need to figure out how to make it stop once the crawler has finished its tasks.
Could anyone provide some guidance on how to make the crawler stop running after it has completed its tasks, so that the node-cron job can create a new instance and start the process again?
Thank you so much for your help
I set up a node-cron job for every 30 seconds, but the problem occurs when the crawler stays open after finishing its tasks. When the cron job arrives, it seems to creates another instance, and the first iteration works fine. However, subsequent iterations do not start scraping all the pages as expected.
In the terminal, it says that the crawler has finished its tasks, but it does not start scraping again. I know that I will eventually need to run it for a longer duration, such as 30 minutes. Therefore, I need to figure out how to make it stop once the crawler has finished its tasks.
Could anyone provide some guidance on how to make the crawler stop running after it has completed its tasks, so that the node-cron job can create a new instance and start the process again?
Thank you so much for your help
