Help regarding request_queue

I am building a website scraper for my users. I want to support upto x no.of child URLs to be scrapable, starting from the startUrl. In somecases, I am seeing duplicate links to be scraped. And in somecases, the no.of urls identified goes into the order of 1000s. I want to control the enqueuing of the urls into the request_queue, to avoid unnecessary costs and duplication of URLs that are being scraped.

Here is my enque function:

const enqueued = await enqueueLinks({
                selector: 'a',
                transformRequestFunction: (request) => {
                    const url = new URL(request.url);

                    // Use just the domain and path as the unique key, ignoring query parameters and hash fragments
                    request.uniqueKey = url.origin + url.pathname;

                    if (request.uniqueKey.includes(startUrls[0].url)) {
                        if (globalStore.capturedLinks.hasOwnProperty(request.uniqueKey)) {
                            return;
                        }
                        globalStore.capturedLinks[request.uniqueKey] = false;
                        return request;
                    }
                },
            });

const enqueued = await enqueueLinks({
                selector: 'a',
                transformRequestFunction: (request) => {
                    const url = new URL(request.url);

                    // Use just the domain and path as the unique key, ignoring query parameters and hash fragments
                    request.uniqueKey = url.origin + url.pathname;

                    if (request.uniqueKey.includes(startUrls[0].url)) {
                        if (globalStore.capturedLinks.hasOwnProperty(request.uniqueKey)) {
                            return;
                        }
                        globalStore.capturedLinks[request.uniqueKey] = false;
                        return request;
                    }
                },
            });

Also, I have set the link selector as aa tag. Should I not use this in the scrape request's input ?

Apify & Crawlee•3y ago•

1 reply

precious-lavender

Help regarding request_queue

const enqueued = await enqueueLinks({
                selector: 'a',
                transformRequestFunction: (request) => {
                    const url = new URL(request.url);

                    // Use just the domain and path as the unique key, ignoring query parameters and hash fragments
                    request.uniqueKey = url.origin + url.pathname;

                    if (request.uniqueKey.includes(startUrls[0].url)) {
                        if (globalStore.capturedLinks.hasOwnProperty(request.uniqueKey)) {
                            return;
                        }
                        globalStore.capturedLinks[request.uniqueKey] = false;
                        return request;
                    }
                },
            });

const enqueued = await enqueueLinks({
                selector: 'a',
                transformRequestFunction: (request) => {
                    const url = new URL(request.url);

                    // Use just the domain and path as the unique key, ignoring query parameters and hash fragments
                    request.uniqueKey = url.origin + url.pathname;

                    if (request.uniqueKey.includes(startUrls[0].url)) {
                        if (globalStore.capturedLinks.hasOwnProperty(request.uniqueKey)) {
                            return;
                        }
                        globalStore.capturedLinks[request.uniqueKey] = false;
                        return request;
                    }
                },
            });

Also, I have set the link selector as aa tag. Should I not use this in the scrape request's input ?

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,851 Members

Join

Help regarding request_queue

Help regarding request_queue

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads