Apify Discord Mirror

Updated 5 months ago

accessing saved data with mongodb in an apofy actor

At a glance

The community member is trying to save data scraped from their actors to a JSON file, but they are not getting the expected JSON output. They want to save the data to the Apify console instead, so they can then access it from the actor, clean the data, and save it to a MongoDB database. The community member has provided some code that reads and writes data to a "rawData.json" file, but they are unsure how to save the data to the Apify console. Other community members have suggested adding syntax highlighting to the code block and providing access to the actor's source code to help troubleshoot the issue.

ive tried saving the data to a rawdata.json file from the data i scrape from my actors, however i dont get a json output even thought the scraping works
how would i save the data to the apify console that i can then use mongodb to take that data and put it in my database - i have my mongodb schema already setup so how would i save the data to the apify console and access it
heres what i have for saving the json file so far:
h
s
5 comments
Plain Text
bambawRouter.addHandler('BAMBAW_PRODUCT', async ({ page, request }) => {
    try {
        console.log('Scraping products');

        const site = 'Bambaw';

        const title = await page.$eval('h1.product__title', (el) => el.textContent?.trim() || '');

        const descriptions = await ......

        const productData = {
        url: request.loadedUrl,
        site,
        title,
        descriptions,
        originalPrice,
        salePrice,
        shippingInfo,
        reviewScore,
        reviewNumber,
        };

        productList.push(productData);

        console.log('Scraped ', productList.length, ' products')
        // Read the existing data from the rawData.json file
        let rawData: any = {};
        try {
            const rawDataStr = fs.readFileSync('rawData.json', 'utf8');
            rawData = JSON.parse(rawDataStr);
        } catch (error) {
            console.log('Error reading rawData.json:', error);
        }

        // Append the new data to the existing data
        if (rawData.productList) {
            rawData.productList.push(productData);
        } else {
            rawData.productList = [productData];
        }

        // Write the updated data back to the rawData.json file
        fs.writeFileSync('rawData.json', JSON.stringify(rawData, null, 2));
        console.log('rawData.json updated for Bambaw');
    } catch (error) {
        console.log('Error scraping product:', error);
        bambawQueue.reclaimRequest(request);
        return;
    }    
});
you can add a language to the three backticks for your code block to give it syntax highlighting–which will help you get help 🙂
i need to be able to save data to the apify console that i can then access from the actor, clean the data, and save to a database like mongodb
if you can get me access to the actor’s source or figure out syntax highlighting here, I can take a look and try to help
If you still need it 🙂
Add a reply
Sign up and join the conversation on Discord