Recover endless pagination items by clicking on showMore button

Hey guys ! Im trying to scrape all the products from a listing with an endless pagination. In order to load the other items, I've got to click on a showMore button. I looked it up on the docs and tried several syntaxes but I couldn't get it to work..

await page.$eval('main', (main) => {
    main.querySelector('.c-endless-paginator__button').click();
})
await page.waitForTimeout(5000);
const products = await page.$$eval('.c-product-list-container-products__item', ($products) => {
    const scrapedUrls = [];
    $products.forEach(async ($product) => {
        await scrapedUrls.push({
            url: $product.querySelector('.c-product-list-item--grid__title').href
        });
    });
    return scrapedUrls;
});
for (let i = 0; i < products.length; i++) {
    await requestQueue.addRequest({
        url: products[i].url,
        userData: {
            depth: request.userData.depth + 1
        }
    })
}

await page.$eval('main', (main) => {
    main.querySelector('.c-endless-paginator__button').click();
})
await page.waitForTimeout(5000);
const products = await page.$$eval('.c-product-list-container-products__item', ($products) => {
    const scrapedUrls = [];
    $products.forEach(async ($product) => {
        await scrapedUrls.push({
            url: $product.querySelector('.c-product-list-item--grid__title').href
        });
    });
    return scrapedUrls;
});
for (let i = 0; i < products.length; i++) {
    await requestQueue.addRequest({
        url: products[i].url,
        userData: {
            depth: request.userData.depth + 1
        }
    })
}

I don't get why the products won't load even after the click and the waitForTimeout. Thank you for your help !

Apify & Crawlee•4y ago•

2 replies

uncertain-scarlet

Recover endless pagination items by clicking on showMore button

await page.$eval('main', (main) => {
    main.querySelector('.c-endless-paginator__button').click();
})
await page.waitForTimeout(5000);
const products = await page.$$eval('.c-product-list-container-products__item', ($products) => {
    const scrapedUrls = [];
    $products.forEach(async ($product) => {
        await scrapedUrls.push({
            url: $product.querySelector('.c-product-list-item--grid__title').href
        });
    });
    return scrapedUrls;
});
for (let i = 0; i < products.length; i++) {
    await requestQueue.addRequest({
        url: products[i].url,
        userData: {
            depth: request.userData.depth + 1
        }
    })
}

await page.$eval('main', (main) => {
    main.querySelector('.c-endless-paginator__button').click();
})
await page.waitForTimeout(5000);
const products = await page.$$eval('.c-product-list-container-products__item', ($products) => {
    const scrapedUrls = [];
    $products.forEach(async ($product) => {
        await scrapedUrls.push({
            url: $product.querySelector('.c-product-list-item--grid__title').href
        });
    });
    return scrapedUrls;
});
for (let i = 0; i < products.length; i++) {
    await requestQueue.addRequest({
        url: products[i].url,
        userData: {
            depth: request.userData.depth + 1
        }
    })
}

I don't get why the products won't load even after the click and the waitForTimeout. Thank you for your help !

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,735 Members

Join

Recover endless pagination items by clicking on showMore button

Recover endless pagination items by clicking on showMore button

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads