I implemented a class for creating CheerioCrawler and adding routers etc. and I extended this class to create specific implementations for various websites. When I run them it finishes after doing amount of max request that I set. Problem is it counts amount of max request for all the instances and stops after that, instead of handling instances seperately.
INFO CheerioCrawler: Starting the crawler.
INFO CheerioCrawler: Crawler reached the maxRequestsPerCrawl limit of 50 requests and will shut down soon. Requests that are in progress will be allowed to finish.
INFO CheerioCrawler: Earlier, the crawler reached the maxRequestsPerCrawl limit of 50 requests and all requests that were in progress at that time have now finished. In total, the crawler processed 60 requests and will shut down.
INFO CheerioCrawler: Final request statistics:
