national-gold•2y ago
Pagination assistance
Hey All,
Hope you are well! I am new here. I am trying to learn web-scraping and for the most part I think i am slowly figuring it out. I have been stuck for days on trying to work out how pagination works... Anyone able to help me? In my example script, I am trying to look for the next button, and if it exists click through it. Not sure what I am doing wrong...
CODE: https://codeshare.io/WLX0X8
4 Replies
genetic-orange•2y ago
Handling pagination | Academy | Apify Documentation
Learn about the three most popular API pagination techniques and how to handle each of them when scraping an API with pagination.
national-goldOP•2y ago
thanks 🙂 ~ think I need to do some more study to understand... do you happen to have any 101 pagination tutorials for dummies haha.
deep-jade•2y ago
That first link and the one below are really all you need
https://docs.apify.com/academy/advanced-web-scraping/scraping-paginated-sites
Scraping paginated sites | Academy | Apify Documentation
Learn how to extract all of a website's listings even if they limit the number of results pages. See code examples for setting up your scraper.
national-goldOP•2y ago
All good, got it working! Ended up revisiting this which seemed to work 🙂 Needed the ChatGPT breakdown to dumb it down for me more hahaa. https://docs.apify.com/academy/puppeteer-playwright/common-use-cases/paginating-through-results
Paginating through results | Academy | Apify Documentation
Learn how to paginate through results on websites that use either page number-based pagination or dynamic lazy-loading pagination.