I have 99 urls in the queue. But scraper finishes crawl after a few urls, why?

The scraper finishes crawl after a few ones everytime. I have 99 urls added to the queue.

This is my config:

I add urls like this:

 const playRequestQ = await RequestQueue.open('q1');
    const urlsToAdd: { url: string; }[] = []
    nodeEvts.on('links', onLinks)

    async function onLinks(link: any;) {
        urlsToAdd.push({ url: `${link}` })
        if (urlsToAdd.length % 99 === 0) {
            await playRequestQ.addRequests(urlsToAdd)
            crawler.run();
        }
    }

 const playRequestQ = await RequestQueue.open('q1');
    const urlsToAdd: { url: string; }[] = []
    nodeEvts.on('links', onLinks)

    async function onLinks(link: any;) {
        urlsToAdd.push({ url: `${link}` })
        if (urlsToAdd.length % 99 === 0) {
            await playRequestQ.addRequests(urlsToAdd)
            crawler.run();
        }
    }

MEE6•3/9/23, 8:29 AM

@Nisthar just advanced to level 2! Thanks for your contributions!

uncertain-scarletOP•3/9/23, 8:29 AM

I can see the urls added inside the queue

HonzaS•3/9/23, 8:32 AM

maybe urls are not unique

uncertain-scarletOP•3/9/23, 9:01 AM

Ok that was the issue i think

uncertain-scarletOP•3/9/23, 9:01 AM

Thanks

military-pink•3/28/23, 6:27 AM

@HonzaS what if I want to scrap the same url again and again ? do I need to re-initialize the crawler ?

Ccurioussoul @HonzaS what if I want to scrap the same url again and again ? do I need to re-i...

HonzaS•3/28/23, 6:35 AM

Then you need to have different uniqueKey for each request. By default uniqueKey is the same as url but you can set your own.
https://crawlee.dev/api/core/class/Request#uniqueKey

military-pink•3/28/23, 6:47 AM

Thankyou very much @HonzaS