Ddead-brown
Maximum urls to crawl from a named request queue
I have created a named request queue, like this:
const namedRequestQueue = await RequestQueue.open('source-name');
Now when the crawler runs, I want it to crawl only 100 urls from this named request.
My objective is to create many named request queues, and execute them one by one and get 100 urls from each request queue.
I cannot use maxRequestsPerCrawl because that will limit the total urls crawled.
How can I do that? maxRequestsPerCrawl is unlimited, but for each requestQueue, crawl only the first 100 urls from the website.
const namedRequestQueue = await RequestQueue.open('source-name');
Now when the crawler runs, I want it to crawl only 100 urls from this named request.
My objective is to create many named request queues, and execute them one by one and get 100 urls from each request queue.
I cannot use maxRequestsPerCrawl because that will limit the total urls crawled.
How can I do that? maxRequestsPerCrawl is unlimited, but for each requestQueue, crawl only the first 100 urls from the website.
