foreign-sapphire
foreign-sapphire17mo ago

Call for help, Tokopedia Scraper too slow and costly?

Hello Apify! I am new to this platform and I have some questions about the Actor that I made using PlawrightCrawler. I'm trying to make scraper for an Indonesian e-shop called Tokopedia, as it was on the ideas page. The scraper searches a keyword on the e-shop, looks at links to product details and then it visits those detail pages and gets data from there. You can see statistics of one run of the script. 1) For searching only one product the usage cost is $0.262, which is a lot in comparison with Crawlers that I used during tutorial. Is that okay when using PlaywrightCrawler? Isn't that too much? 2) Scraping only one product took 9 m 30 s, which is - again - a lot in comparison with Crawlers that I used during tutorial. Am I doing something wrong? Should I use different type of Crawler or something? 3) I wrote email to ideas@apify.com that I am interested in bringing an idea to life (Tokopedia Scraper), but nobody has responded for days. Is that okay or should I worry about any of my settings being wrong. Thank you in advance for your help <3 I really love your platform!
No description
2 Replies
Oleg V.
Oleg V.17mo ago
Hi. Yeah, looks not very efficient. I guess, in general PlawrightCrawler is a bad decision. Most probably it should work with plainf http (Cheerio / Basic crawlers) If it's e-commerce, possibly there is some XHR request / page variable (e.q. NEXT_DATA) with all needed info. Try to make some research. You can check https://docs.apify.com/academy for some inspiration. This is a nice article in your case too: https://blog.apify.com/web-scraping-in-2018-forget-html-use-xhrs-metadata-or-javascript-variables-8167f252439c/
foreign-sapphire
foreign-sapphireOP17mo ago
Thank you very much for your response and for these materials. I'm looking forward to read them 😁

Did you find this page helpful?