precious-lavenderP
Apify & Crawlee•3y ago•
1 reply
precious-lavender

Help regarding request_queue

I am building a website scraper for my users. I want to support upto x no.of child URLs to be scrapable, starting from the startUrl. In somecases, I am seeing duplicate links to be scraped. And in somecases, the no.of urls identified goes into the order of 1000s. I want to control the enqueuing of the urls into the request_queue, to avoid unnecessary costs and duplication of URLs that are being scraped.

Here is my enque function:
const enqueued = await enqueueLinks({
                selector: 'a',
                transformRequestFunction: (request) => {
                    const url = new URL(request.url);

                    // Use just the domain and path as the unique key, ignoring query parameters and hash fragments
                    request.uniqueKey = url.origin + url.pathname;

                    if (request.uniqueKey.includes(startUrls[0].url)) {
                        if (globalStore.capturedLinks.hasOwnProperty(request.uniqueKey)) {
                            return;
                        }
                        globalStore.capturedLinks[request.uniqueKey] = false;
                        return request;
                    }
                },
            });


Also, I have set the link selector as a tag. Should I not use this in the scrape request's input ?
Was this page helpful?