Apify Discord Mirror

Updated 5 months ago

Scraping Threads Timeline. But Not able to get entire data.

At a glance

The community member is creating a scraper using Playwright and Python to scrape threads.net, but is unable to scrape the entire timeline as the new data is not being scraped when scrolling down. The comments suggest trying auto-scrolling or finding site-specific controls, as well as storing screenshots to analyze where the issue is occurring. One community member indicates that the suggestions helped.

Useful resources

SSoul_Denied

I am making a scrapper to scrap threads.net using playwright and python and willing to host on Apify.

But It I am not able to scrap entire timeline. It is only giving some data and when scrolling down, I am not able to scrap the new data.

Any help or suggestion is highly appreciated.

3 comments

AAlexey Udovydchenko

Try https://docs.apify.com/academy/puppeteer-playwright/common-use-cases/paginating-through-results#auto-scrolling or find site-specific controls

LLukas Krivka

I recommend storing screenshots often, you will see exactly where you got stuck and why. More here https://docs.apify.com/academy/node-js/analyzing-pages-and-fixing-errors

SSoul_Denied

Thank you guys. It helped.

Add a reply