saving data and accessing it in an Apify Actor

At a glance

The community member is trying to save scraped data from their actors to a rawdata.json file, but they are not getting a JSON output. They want to know how to save the data to the Apify console so they can then use MongoDB to store the data in their database. The community member has provided some code that reads from the rawdata.json file, appends the new data, and writes the updated data back to the file.

hharish

ive tried saving the data to a rawdata.json file from the data i scrape from my actors, however i dont get a json output even thought the scraping works
how would i save the data to the apify console that i can then use mongodb to take that data and put it in my database - i have my mongodb schema already setup so how would i save the data to the apify console and access it
heres what i have for saving the json file so far:

1 comment

hharish

Plain Text

bambawRouter.addHandler('BAMBAW_PRODUCT', async ({ page, request }) => {
    try {
        console.log('Scraping products');

        const site = 'Bambaw';

        const title = await page.$eval('h1.product__title', (el) => el.textContent?.trim() || '');

        const descriptions = await ......

        const productData = {
        url: request.loadedUrl,
        site,
        title,
        descriptions,
        originalPrice,
        salePrice,
        shippingInfo,
        reviewScore,
        reviewNumber,
        };

        productList.push(productData);

        console.log('Scraped ', productList.length, ' products')
        // Read the existing data from the rawData.json file
        let rawData: any = {};
        try {
            const rawDataStr = fs.readFileSync('rawData.json', 'utf8');
            rawData = JSON.parse(rawDataStr);
        } catch (error) {
            console.log('Error reading rawData.json:', error);
        }

        // Append the new data to the existing data
        if (rawData.productList) {
            rawData.productList.push(productData);
        } else {
            rawData.productList = [productData];
        }

        // Write the updated data back to the rawData.json file
        fs.writeFileSync('rawData.json', JSON.stringify(rawData, null, 2));
        console.log('rawData.json updated for Bambaw');
    } catch (error) {
        console.log('Error scraping product:', error);
        bambawQueue.reclaimRequest(request);
        return;
    }

Add a reply

Apify Discord Mirror

saving data and accessing it in an Apify Actor