evident-indigo•3y ago
Google play reviews scraper
Can anyone look into the code below, I don't know what's wrong but i can't scrape reviews from play.google.com using below code.
7 Replies
evident-indigoOP•3y ago
async requestHandler({ request, page, enqueueLinks }) {
console.log(Scraping ${request.url}...);
console.log('New page created')
await page.waitForTimeout(1000);
await page.click("button[jscontroller='soHxf']");
let pageData = await page.evaluate(
() => document.querySelector("*").outerHTML
const $ = cheerio.load(pageData);
const data = [];
$(".EGFGHd").each((i, el) => {
let reviewDate = $(el).find("span.bp9Aid").text();
let reviewAuthor = $(el).find("div.X5PpBb").text();
let reviewTitle = $(el).find("").text();
let reviewDesc = $(el).find(".h3YV2d").text();
let overallRatings = $(el).find(".iXRFPc").attr("aria-label").split(" ")[1];
data.push({
author: reviewAuthor,
date: reviewDate,
sourceCollector: 'play.google.com',
sourceURL: request.url,
title: reviewTitle,
description: reviewDesc,
ratings: overallRatings
});
});
await Actor.pushData(data);
}evident-indigoOP•3y ago
i need to scrape all the reviews for this app https://play.google.com/store/apps/details?id=com.spoken.app&hl=en_US&gl=US&pli=1
Spoken – Tap to Talk AAC - Apps on Google Play
Speak naturally with powerful text-to-speech voice technology
foreign-sapphire•3y ago
Hi @thenameispriyam, can you please send me the failed run link in a private message?
evident-indigoOP•3y ago
yeah sure
foreign-sapphire•3y ago
You have correct review wrapper selector here (.EGFGHd), but wrong selector in the run you have sent me.
evident-indigoOP•3y ago
I'm getting only 3 reviews using above selector which are shown on the homepage of the app, but i need to scrape all reviews which requires using the selector in the run.
foreign-sapphire•3y ago
Then the issue is probably in the button clicking action. You can try to add some waiting and test the scraper locally with headful mode.