Alex I•15mo ago

How can I run an Actor local with Actor.config inputs?

I'd like run the next code localy

// Initialize the Apify SDK
await Actor.init();
const privateKey: string = Actor.config.get('private_key');
const clientEmail: string = Actor.config.get('client_email');

// Initialize the Apify SDK
await Actor.init();
const privateKey: string = Actor.config.get('private_key');
const clientEmail: string = Actor.config.get('client_email');

But privateKey is undefined I expected to use it like npx apify-cli run but nothing. Any suggestion?

2 Replies

Alex IOP•15mo ago

To run an Actor locally and read from Actor.config, you need to ensure your local environment mimics the remote environment where the Actor.config is available. The Actor.config file is typically used in the Apify platform, so running it locally requires a few extra steps to simulate this environment. Here's how you can set it up: 1. Create Actor.config Locally: Create a local Actor.config file in the root directory of your project with the required inputs. For example:

   {
       "startUrl": "https://example.com"
   }

   {
       "startUrl": "https://example.com"
   }

2. Set Up Local Environment: Create a local script to read this configuration file and use it as the input for your actor. 3. Modify the Script to Read Local Configuration: Adjust your script to read the configuration file locally when not running on the Apify platform. Here's an updated script that can run locally:

import { PlaywrightCrawler } from 'crawlee';
import { Actor } from 'apify';
import fs from 'fs';

(async () => {
    // Initialize the Actor
    await Actor.init();

    // Read the configuration from Actor.config
    let input;
    if (process.env.NODE_ENV === 'production') {
        input = await Actor.getInput();
    } else {
        // Read the local Actor.config file
        const configPath = './Actor.config';
        if (fs.existsSync(configPath)) {
            input = JSON.parse(fs.readFileSync(configPath, 'utf-8'));
        } else {
            console.error('Local Actor.config file not found.');
            process.exit(1);
        }
    }
    console.log('Input configuration:', input);

    // Create a PlaywrightCrawler
    const crawler = new PlaywrightCrawler({
        requestHandler: async ({ request, page, enqueueLinks, pushData, log }) => {
            const title = await page.title();
            log.info(`Title of ${request.loadedUrl} is '${title}'`);

            // Save results as JSON
            await pushData({ title, url: request.loadedUrl });

            // Extract links and add them to the queue
            await enqueueLinks();
        },
        headless: true,
        maxRequestsPerCrawl: 20,
    });

    // Add the start URL from the input configuration and start the crawl
    await crawler.run([input.startUrl]);

    // Clean up and exit the Actor
    await Actor.exit();
})();

import { PlaywrightCrawler } from 'crawlee';
import { Actor } from 'apify';
import fs from 'fs';

(async () => {
    // Initialize the Actor
    await Actor.init();

    // Read the configuration from Actor.config
    let input;
    if (process.env.NODE_ENV === 'production') {
        input = await Actor.getInput();
    } else {
        // Read the local Actor.config file
        const configPath = './Actor.config';
        if (fs.existsSync(configPath)) {
            input = JSON.parse(fs.readFileSync(configPath, 'utf-8'));
        } else {
            console.error('Local Actor.config file not found.');
            process.exit(1);
        }
    }
    console.log('Input configuration:', input);

    // Create a PlaywrightCrawler
    const crawler = new PlaywrightCrawler({
        requestHandler: async ({ request, page, enqueueLinks, pushData, log }) => {
            const title = await page.title();
            log.info(`Title of ${request.loadedUrl} is '${title}'`);

            // Save results as JSON
            await pushData({ title, url: request.loadedUrl });

            // Extract links and add them to the queue
            await enqueueLinks();
        },
        headless: true,
        maxRequestsPerCrawl: 20,
    });

    // Add the start URL from the input configuration and start the crawl
    await crawler.run([input.startUrl]);

    // Clean up and exit the Actor
    await Actor.exit();
})();

Step-by-Step Explanation: 1. Environment Detection: - The script checks if it's running in a production environment (assuming NODE_ENV is set to production on the Apify platform). If not, it reads the Actor.config file locally.
2. Local Configuration Reading: - When running locally, the script reads the Actor.config file from the project's root directory using Node.js's fs module. 3. Crawler Setup and Execution: - The rest of the script remains the same, setting up and running the PlaywrightCrawler with the configuration read from the input. Notes: - Ensure your Actor.config file is correctly formatted and available in the root directory of your project when running locally. - You can adjust the logic to handle different configurations and input structures as needed. - Remember to install the necessary dependencies (crawlee, apify, and fs module if not already available). This approach should allow you to run your actor both locally and remotely with the same codebase, adjusting only the way the configuration is read. Is this optimal approach?

Pepa J•15mo ago

Hi @Alex I I am little bit confused, seems to me that you use the term config in the same way as Input You may create the file INPUT.json locally as described here https://discord.com/channels/801163717915574323/1252904017240002560/1252904017240002560 Also if you generate your project via crawlee-cli and run it via npx @crawlee/cli it should create this folder/input structure automatically for you.

How can I run an Actor local with Actor.config inputs?

Did you find this page helpful?