Hello! need to scrape multiple links within same page using Puppeteer

The title of my post is not explicit , in fact I am scraping a website with multiple products but the category tree of the menu is a little complex, There are products category on the top level and products families on the sub-level, I need to retrieve a count of the products of each category but the only way to do that is to go through each family of product. I am stuck at this point how do I enqueue Links within the category products page ( there are more than one families within a category );
IN ADDITION I need to push the name of the category in the dataset, AND the name of each family as such:

{ "category" { "family" : "family_name", "detail": 10}}

{ "category" { "family" : "family_name", "detail": 10}}

Here is my code :

    await enqueueLinks({
        globs:['https://www.1fotrade.com/pro/categorie-*'],
        selector:'td.element_menu a',
        label:'list'
     })
    
}); 

router.addHandler('list', async ({ enqueueLinks, request, page, log }) => {
log.debug(`Extracting data: ${request.url}`);    
const category = await page.$eval('h1',el=> el.textContent); 

await enqueueLinks({
        globs:['https://www.1fotrade.com/pro/famille-*'],
        selector:'.menu_titre_famille a',
        label: 'detail',
    });
 await Dataset.pushData({      
                 category
    });

});

router.addHandler( 'detail',async ({ request, page, log }) => {    
    const title = await page.title();
    log.info(`${title}`, { url: request.loadedUrl });
 const items  = await page.$$('.liste_produit tr.productLine');
 const detail = items.length;
 const name  = await page.$eval('h1',el => el.textContent);
  

 await Dataset.pushData({      
                 name, detail
    });
});

    await enqueueLinks({
        globs:['https://www.1fotrade.com/pro/categorie-*'],
        selector:'td.element_menu a',
        label:'list'
     })
    
}); 

router.addHandler('list', async ({ enqueueLinks, request, page, log }) => {
log.debug(`Extracting data: ${request.url}`);    
const category = await page.$eval('h1',el=> el.textContent); 

await enqueueLinks({
        globs:['https://www.1fotrade.com/pro/famille-*'],
        selector:'.menu_titre_famille a',
        label: 'detail',
    });
 await Dataset.pushData({      
                 category
    });

});

router.addHandler( 'detail',async ({ request, page, log }) => {    
    const title = await page.title();
    log.info(`${title}`, { url: request.loadedUrl });
 const items  = await page.$$('.liste_produit tr.productLine');
 const detail = items.length;
 const name  = await page.$eval('h1',el => el.textContent);
  

 await Dataset.pushData({      
                 name, detail
    });
});

Apify & Crawlee•4y ago•

1 reply

hurt-tomato

Hello! need to scrape multiple links within same page using Puppeteer

{ "category" { "family" : "family_name", "detail": 10}}

{ "category" { "family" : "family_name", "detail": 10}}

Here is my code :

    await enqueueLinks({
        globs:['https://www.1fotrade.com/pro/categorie-*'],
        selector:'td.element_menu a',
        label:'list'
     })
    
}); 

router.addHandler('list', async ({ enqueueLinks, request, page, log }) => {
log.debug(`Extracting data: ${request.url}`);    
const category = await page.$eval('h1',el=> el.textContent); 

await enqueueLinks({
        globs:['https://www.1fotrade.com/pro/famille-*'],
        selector:'.menu_titre_famille a',
        label: 'detail',
    });
 await Dataset.pushData({      
                 category
    });

});

router.addHandler( 'detail',async ({ request, page, log }) => {    
    const title = await page.title();
    log.info(`${title}`, { url: request.loadedUrl });
 const items  = await page.$$('.liste_produit tr.productLine');
 const detail = items.length;
 const name  = await page.$eval('h1',el => el.textContent);
  

 await Dataset.pushData({      
                 name, detail
    });
});

    await enqueueLinks({
        globs:['https://www.1fotrade.com/pro/categorie-*'],
        selector:'td.element_menu a',
        label:'list'
     })
    
}); 

router.addHandler('list', async ({ enqueueLinks, request, page, log }) => {
log.debug(`Extracting data: ${request.url}`);    
const category = await page.$eval('h1',el=> el.textContent); 

await enqueueLinks({
        globs:['https://www.1fotrade.com/pro/famille-*'],
        selector:'.menu_titre_famille a',
        label: 'detail',
    });
 await Dataset.pushData({      
                 category
    });

});

router.addHandler( 'detail',async ({ request, page, log }) => {    
    const title = await page.title();
    log.info(`${title}`, { url: request.loadedUrl });
 const items  = await page.$$('.liste_produit tr.productLine');
 const detail = items.length;
 const name  = await page.$eval('h1',el => el.textContent);
  

 await Dataset.pushData({      
                 name, detail
    });
});

Hello! need to scrape multiple links within same page using Puppeteer

Similar Threads

Hello! need to scrape multiple links within same page using Puppeteer

Similar Threads

Similar Threads

Similar Threads