2 replies

Recover endless pagination items by clicking on showMore button

Hey guys ! Im trying to scrape all the products from a listing with an endless pagination. In order to load the other items, I've got to click on a showMore button. I looked it up on the docs and tried several syntaxes but I couldn't get it to work..

await page.$eval('main', (main) => {
    main.querySelector('.c-endless-paginator__button').click();
})
await page.waitForTimeout(5000);
const products = await page.$$eval('.c-product-list-container-products__item', ($products) => {
    const scrapedUrls = [];
    $products.forEach(async ($product) => {
        await scrapedUrls.push({
            url: $product.querySelector('.c-product-list-item--grid__title').href
        });
    });
    return scrapedUrls;
});
for (let i = 0; i < products.length; i++) {
    await requestQueue.addRequest({
        url: products[i].url,
        userData: {
            depth: request.userData.depth + 1
        }
    })
}

await page.$eval('main', (main) => {
    main.querySelector('.c-endless-paginator__button').click();
})
await page.waitForTimeout(5000);
const products = await page.$$eval('.c-product-list-container-products__item', ($products) => {
    const scrapedUrls = [];
    $products.forEach(async ($product) => {
        await scrapedUrls.push({
            url: $product.querySelector('.c-product-list-item--grid__title').href
        });
    });
    return scrapedUrls;
});
for (let i = 0; i < products.length; i++) {
    await requestQueue.addRequest({
        url: products[i].url,
        userData: {
            depth: request.userData.depth + 1
        }
    })
}

I don't get why the products won't load even after the click and the waitForTimeout. Thank you for your help !

Recover endless pagination items by clicking on showMore button

Similar Threads