Waiting for all requests to be added before hitting a request handler
Hi!
I have two requestsHandlers, registered from my router. Lets call them SITEMAP and PRODUCT.
I get all of the URLs from the sitemap.xml and push them into a queue with addRequestsBatched, and i set the wait waitForAllRequestsToBeAdded to true. My urls have the "PRODUCT" label, so they will hit the "PRODUCT" route.
The problem is, that even though I have 30k requests, as soon as the first batch of 1000 is added, my "PRODUCT" router stars crawling them.
Is there any posibility of not letting that route to handle the requests, until the queue loads all of the URLs ?
What I am trying to achieve with that, is if my node process crashes for example, and I managed to load all of the URLs, the crawling from the "PRODUCTS" route will resume seamesly, as it already has all of the URLs from the sitemap already loaded.
Thanks!
I have two requestsHandlers, registered from my router. Lets call them SITEMAP and PRODUCT.
I get all of the URLs from the sitemap.xml and push them into a queue with addRequestsBatched, and i set the wait waitForAllRequestsToBeAdded to true. My urls have the "PRODUCT" label, so they will hit the "PRODUCT" route.
The problem is, that even though I have 30k requests, as soon as the first batch of 1000 is added, my "PRODUCT" router stars crawling them.
Is there any posibility of not letting that route to handle the requests, until the queue loads all of the URLs ?
What I am trying to achieve with that, is if my node process crashes for example, and I managed to load all of the URLs, the crawling from the "PRODUCTS" route will resume seamesly, as it already has all of the URLs from the sitemap already loaded.
Thanks!
