Apify & CrawleeA&CApify & Crawlee
Powered by
embarrassing-maroonE
Apify & Crawlee•3y ago•
3 replies
embarrassing-maroon

conducting faster scrapes with pagination and individual product scraping

hey i was curious that when im scraping amazon, what's a reasonable time frame for the scraping duration considering i scrape each product link from the results page and then scrape each individual product page for the information and also paginate through each results page until there are no more pages left
i did previously just scrape product info straight of product cards on the results page but it would some times give dummy links that would lead to an unrelated amazon page and the product info would be more innacurate
how can i increase the speed of my scrapes, especially considering i want add on more and more scrapers in the future that i all want to happen concurrently to save time, im aiming for quite a low scrape time of within 10 seconds - 15 seconds or lower and its taking upwards of 1 minute
this is a cheerio crawler
Apify & Crawlee banner
Apify & CrawleeJoin
This is the official developer community of Apify and Crawlee.
13,739Members
Resources

Similar Threads

Was this page helpful?
Recent Announcements
Recent Announcements
ellativity

**The Apify $1M Challenge is over!** For everyone who joined yesterday’s Award Ceremony livestream for the Apify $1M Challenge, thank you for your enthusiastic drumrolls in the chat and positive vibes. We were really feeling the excitement and celebratory mood! If you missed the stream or just want to rewatch the key moments again, here’s the replay link https://www.youtube.com/watch?v=eEDV-5X43Gg (ngl, the replay is not the same without your live chat) And, if you didn’t check the email that should have landed in your inboxes, we’d love to hear about your experience of participating in the Apify $1M Challenge. **<a:alerthulk:1468892073917939713> Win one of five $100 Visa gift cards by completing the end-of-challenge survey here: https://apify.typeform.com/to/mjoMaZqD** Thank you again to everyone who participated in any capacity. The past 3 months have been a wild ride and we feel so grateful to have been on this adventure with y’all. We mean every word when we say how much you’ve impressed us. Thank you all from the bottom of our hearts. <a:keanuthanks:1430839059655426068> Saurav and Ella xoxo PS - if you just want to jump to the spoilers, a full list of winners is available at https://apify.com/challenge 🏆

ellativity · 5d ago

ellativity

**You are invited** ... to celebrate all the achievements of the Apify $1M Challenge with us on Wednesday, February 4 at **8 AM PT / 11 AM ET / 4 PM GMT / 5 PM CET / 9:30 PM IST / 12 AM +1d CST** We will be announcing winners of the Grand Prizes, as well as regional winners and much more, with especially good news for all participating developers. 🏆 We look forward to sharing with you all! 🎉 More info here: https://luma.com/6c1493t0

ellativity · 2w ago

ellativity

Hi @everyone 👋 I'm hanging out in https://discord.com/channels/801163717915574323/1430491198145167371 for the next 45 min, if you want to discuss the end of the challenge or anything else.

ellativity · 2w ago

Similar Threads

taking list of scraped urls and conducting multiple new scrapes
embarrassing-maroonEembarrassing-maroon / crawlee-js
3y ago
enqueueLinks with pagination
CasperCCasper / crawlee-js
4y ago
CheerioCrawler only scrapes one url
faint-whiteFfaint-white / crawlee-js
3y ago
Scraping Lazada.
ordinary-sapphireOordinary-sapphire / crawlee-js
3y ago