enqueueLinks works without options but with options it does not work

Hi everyone I was trying just for fun to crawl the idealista page for some homes. While doing so I noticed that enqueueLinks() without options works but as soon as I put a label and a selector it does not select anything and stops. Any idea what it could be?

StartUrl example: https://www.idealista.com/venta-viviendas/barcelona/sant-marti/

requestHandler: async ({page, log, request,enqueueLinks}) =>{
        if(request.label === 'LIST'){
            await sleep(5000);
        }else if (request.label === 'HOME'){
            await sleep(5000);
        }else {
            log.info(`Processing: ${request.url}`);
            // Check if Cookie concent is there.
            await page.waitForSelector('#didomi-notice-agree-button');
            const cookieAcceptButton = await page.$('#didomi-notice-agree-button');
            if (cookieAcceptButton) {
                await page.click('#didomi-notice-agree-button');
            }
            
            // Check if there are more pages to que them up
            await page.waitForLoadState('networkidle');
            await page.waitForSelector('li.next');
            const nextButton = await page.$('li.next');
            if (nextButton) {
                await enqueueLinks({
                    selector: 'li.next',
                    strategy: 'all',
                    label: 'LIST',
                });
            }
        }
    },

requestHandler: async ({page, log, request,enqueueLinks}) =>{
        if(request.label === 'LIST'){
            await sleep(5000);
        }else if (request.label === 'HOME'){
            await sleep(5000);
        }else {
            log.info(`Processing: ${request.url}`);
            // Check if Cookie concent is there.
            await page.waitForSelector('#didomi-notice-agree-button');
            const cookieAcceptButton = await page.$('#didomi-notice-agree-button');
            if (cookieAcceptButton) {
                await page.click('#didomi-notice-agree-button');
            }
            
            // Check if there are more pages to que them up
            await page.waitForLoadState('networkidle');
            await page.waitForSelector('li.next');
            const nextButton = await page.$('li.next');
            if (nextButton) {
                await enqueueLinks({
                    selector: 'li.next',
                    strategy: 'all',
                    label: 'LIST',
                });
            }
        }
    },

Apify & Crawlee•3y ago•

1 reply

verbal-lime

enqueueLinks works without options but with options it does not work

requestHandler: async ({page, log, request,enqueueLinks}) =>{
        if(request.label === 'LIST'){
            await sleep(5000);
        }else if (request.label === 'HOME'){
            await sleep(5000);
        }else {
            log.info(`Processing: ${request.url}`);
            // Check if Cookie concent is there.
            await page.waitForSelector('#didomi-notice-agree-button');
            const cookieAcceptButton = await page.$('#didomi-notice-agree-button');
            if (cookieAcceptButton) {
                await page.click('#didomi-notice-agree-button');
            }
            
            // Check if there are more pages to que them up
            await page.waitForLoadState('networkidle');
            await page.waitForSelector('li.next');
            const nextButton = await page.$('li.next');
            if (nextButton) {
                await enqueueLinks({
                    selector: 'li.next',
                    strategy: 'all',
                    label: 'LIST',
                });
            }
        }
    },

requestHandler: async ({page, log, request,enqueueLinks}) =>{
        if(request.label === 'LIST'){
            await sleep(5000);
        }else if (request.label === 'HOME'){
            await sleep(5000);
        }else {
            log.info(`Processing: ${request.url}`);
            // Check if Cookie concent is there.
            await page.waitForSelector('#didomi-notice-agree-button');
            const cookieAcceptButton = await page.$('#didomi-notice-agree-button');
            if (cookieAcceptButton) {
                await page.click('#didomi-notice-agree-button');
            }
            
            // Check if there are more pages to que them up
            await page.waitForLoadState('networkidle');
            await page.waitForSelector('li.next');
            const nextButton = await page.$('li.next');
            if (nextButton) {
                await enqueueLinks({
                    selector: 'li.next',
                    strategy: 'all',
                    label: 'LIST',
                });
            }
        }
    },

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,851 Members

Join

enqueueLinks works without options but with options it does not work

enqueueLinks works without options but with options it does not work

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads