2 replies

infiniteScroll and enqueueLinksByClickingElements

Hello Team,

I'm trying to crawl a page that has lazy loaded images (on scroll) and an element on the first page that is a JS event 'button' that expands the pages of "posts" on the page.

I'm trying to use the below code, however, it seems like the request never gets filled, the stats show 'requestsTotal:0'.

async requestHandler({ request, page, enqueueLinks, enqueueLinksByClickingElements, infiniteScroll, log }) {

    // Extract links from the current page
    // and add them to the crawling queue.
    await enqueueLinks({ 
      ...<snip>
    });

    await enqueueLinksByClickingElements({
      page,
      selector: '.js-page-load',
      requestQueue: rQueue,
    });

    await infiniteScroll(page, { timeoutSecs: 3000, waitForSecs: 1000 });
    
}

async requestHandler({ request, page, enqueueLinks, enqueueLinksByClickingElements, infiniteScroll, log }) {

    // Extract links from the current page
    // and add them to the crawling queue.
    await enqueueLinks({ 
      ...<snip>
    });

    await enqueueLinksByClickingElements({
      page,
      selector: '.js-page-load',
      requestQueue: rQueue,
    });

    await infiniteScroll(page, { timeoutSecs: 3000, waitForSecs: 1000 });
    
}

I'm trying to target this:

<button class="pagination-load js-page-load" data-href="/page/2/">Load More <span></span></button>

<button class="pagination-load js-page-load" data-href="/page/2/">Load More <span></span></button>

On the first page..so far it seems like this button is only on the first page.

I'm also using preNavigationHooks to read the network requests and store image LINKS only. I don't know if this code should be in preNavigation hooks instead? Not sure. Thanks for your help as always.

Apify & Crawlee•3y ago•

2 replies

cryptorex

infiniteScroll and enqueueLinksByClickingElements

async requestHandler({ request, page, enqueueLinks, enqueueLinksByClickingElements, infiniteScroll, log }) {

    // Extract links from the current page
    // and add them to the crawling queue.
    await enqueueLinks({ 
      ...<snip>
    });

    await enqueueLinksByClickingElements({
      page,
      selector: '.js-page-load',
      requestQueue: rQueue,
    });

    await infiniteScroll(page, { timeoutSecs: 3000, waitForSecs: 1000 });
    
}

async requestHandler({ request, page, enqueueLinks, enqueueLinksByClickingElements, infiniteScroll, log }) {

    // Extract links from the current page
    // and add them to the crawling queue.
    await enqueueLinks({ 
      ...<snip>
    });

    await enqueueLinksByClickingElements({
      page,
      selector: '.js-page-load',
      requestQueue: rQueue,
    });

    await infiniteScroll(page, { timeoutSecs: 3000, waitForSecs: 1000 });
    
}

I'm trying to target this:

<button class="pagination-load js-page-load" data-href="/page/2/">Load More <span></span></button>

<button class="pagination-load js-page-load" data-href="/page/2/">Load More <span></span></button>

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

15,108 Members

Join

infiniteScroll and enqueueLinksByClickingElements

infiniteScroll and enqueueLinksByClickingElements

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads