Hello! need to scrape multiple links within same page using Puppeteer

The title of my post is not explicit , in fact I am scraping a website with multiple products but the category tree of the menu is a little complex, There are products category on the top level and products families on the sub-level, I need to retrieve a count of the products of each category but the only way to do that is to go through each family of product. I am stuck at this point how do I enqueue Links within the category products page ( there are more than one families within a category );
IN ADDITION I need to push the name of the category in the dataset, AND the name of each family as such:
{ "category" { "family" : "family_name", "detail": 10}} { "category" { "family" : "family_name", "detail": 10}}

Here is my code :

    await enqueueLinks({
        globs:['https://www.1fotrade.com/pro/categorie-*'],
        selector:'td.element_menu a',
        label:'list'
     })
    
}); 

router.addHandler('list', async ({ enqueueLinks, request, page, log }) => {
log.debug(`Extracting data: ${request.url}`);    
const category = await page.$eval('h1',el=> el.textContent); 

await enqueueLinks({
        globs:['https://www.1fotrade.com/pro/famille-*'],
        selector:'.menu_titre_famille a',
        label: 'detail',
    });
 await Dataset.pushData({      
                 category
    });

});

router.addHandler( 'detail',async ({ request, page, log }) => {    
    const title = await page.title();
    log.info(`${title}`, { url: request.loadedUrl });
 const items  = await page.$$('.liste_produit tr.productLine');
 const detail = items.length;
 const name  = await page.$eval('h1',el => el.textContent);
  

 await Dataset.pushData({      
                 name, detail
    });
});

    await enqueueLinks({
        globs:['https://www.1fotrade.com/pro/categorie-*'],
        selector:'td.element_menu a',
        label:'list'
     })
    
}); 

router.addHandler('list', async ({ enqueueLinks, request, page, log }) => {
log.debug(`Extracting data: ${request.url}`);    
const category = await page.$eval('h1',el=> el.textContent); 

await enqueueLinks({
        globs:['https://www.1fotrade.com/pro/famille-*'],
        selector:'.menu_titre_famille a',
        label: 'detail',
    });
 await Dataset.pushData({      
                 category
    });

});

router.addHandler( 'detail',async ({ request, page, log }) => {    
    const title = await page.title();
    log.info(`${title}`, { url: request.loadedUrl });
 const items  = await page.$$('.liste_produit tr.productLine');
 const detail = items.length;
 const name  = await page.$eval('h1',el => el.textContent);
  

 await Dataset.pushData({      
                 name, detail
    });
});

Apify & Crawlee•4y ago•

1 reply

hurt-tomato

Hello! need to scrape multiple links within same page using Puppeteer

    await enqueueLinks({
        globs:['https://www.1fotrade.com/pro/categorie-*'],
        selector:'td.element_menu a',
        label:'list'
     })
    
}); 

router.addHandler('list', async ({ enqueueLinks, request, page, log }) => {
log.debug(`Extracting data: ${request.url}`);    
const category = await page.$eval('h1',el=> el.textContent); 

await enqueueLinks({
        globs:['https://www.1fotrade.com/pro/famille-*'],
        selector:'.menu_titre_famille a',
        label: 'detail',
    });
 await Dataset.pushData({      
                 category
    });

});

router.addHandler( 'detail',async ({ request, page, log }) => {    
    const title = await page.title();
    log.info(`${title}`, { url: request.loadedUrl });
 const items  = await page.$$('.liste_produit tr.productLine');
 const detail = items.length;
 const name  = await page.$eval('h1',el => el.textContent);
  

 await Dataset.pushData({      
                 name, detail
    });
});

    await enqueueLinks({
        globs:['https://www.1fotrade.com/pro/categorie-*'],
        selector:'td.element_menu a',
        label:'list'
     })
    
}); 

router.addHandler('list', async ({ enqueueLinks, request, page, log }) => {
log.debug(`Extracting data: ${request.url}`);    
const category = await page.$eval('h1',el=> el.textContent); 

await enqueueLinks({
        globs:['https://www.1fotrade.com/pro/famille-*'],
        selector:'.menu_titre_famille a',
        label: 'detail',
    });
 await Dataset.pushData({      
                 category
    });

});

router.addHandler( 'detail',async ({ request, page, log }) => {    
    const title = await page.title();
    log.info(`${title}`, { url: request.loadedUrl });
 const items  = await page.$$('.liste_produit tr.productLine');
 const detail = items.length;
 const name  = await page.$eval('h1',el => el.textContent);
  

 await Dataset.pushData({      
                 name, detail
    });
});

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,735 Members

Join

Hello! need to scrape multiple links within same page using Puppeteer

Hello! need to scrape multiple links within same page using Puppeteer

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads