ripe-grayR
Apify & Crawleeโ€ข2y agoโ€ข
4 replies
ripe-gray

How to determine if dynamic content is loaded or not. PuppeteerCrawler

in the requestHandler I'm trying to click to the pagination next button and I cannot determine if the content is changed or not.
How can I do it? waitfornetworkidle does not seem to work here. any ideas? See the GIF

new PuppeteerCrawler({
preNavigationHooks: [
        async ({ page }) => {
            page.on('response', async (res) => {
                if (res.url().includes('api/offersearches/filters')) {
                    try {
                        const json = await res.json();
                        const jsonString = JSON.stringify(json);
                        const filePath = 'data.json';
                        fs.appendFile(filePath, jsonString + '\n', () => {});
                    } catch (err) {
                        console.error('Response wasn\'t JSON or failed to parse response.');
                    }
                }
            });
        },
    ],
    async requestHandler({ request, page }) {
        for (let i = 0; i < maxNumberOfPages; i++) {
            const isDisabled = await page.evaluate(() => document.querySelector('[data-testid="mo-pagination-next"] button.mo-button--pagination').disabled);
            if (isDisabled) {
                break;
            }

            await Promise.all([
                page.waitForNetworkIdle(),
                page.click('[data-testid="mo-pagination-next"] button.mo-button--pagination'),
            ]);
            console.log('clicked'); // it never reaches
        }
    },
});


Here's my code so far. Currently button is clicked OK, the data is fetched OK. it just hangs in the end, I guess waitForNetworkIdle is never resolving
screencast_2024-05-28_12-08-08.gif
Was this page helpful?