How to call apify actors from a separate server?

Context:
- I'm currently using playwright on my nextjs api routes
- these playwright scripts scrape a website & persist the data in my database (postgres)
- since I need IP roration with session management though, I'd love to offload the scraping to crawlee
- I'm considering apify as the deployment platform as this seems to be the recommended setup and apify apparently also supports webhooks -- e.g. can I get notified about the status of my actor run, such as job.startedjob.started or job.finishedjob.finished events?
- I'd love for the crawlee scraper on apify to persist data to my database so that my nextjs api refers to the database as the source of truth

I imagine I'd call something like this within my nextjs api route:

import { ApifyClient } from "apify-client"
const apify = new ApifyClient({ token: 'my-token' });


// Resource clients accept an ID of the resource.
const actor = apify.actor('john-doe/my-actor');
// Fetches the john-doe/my-actor object from the API.
const myActor = await actor.get();
// Starts the run of john-doe/my-actor and returns the Run object.
const myActorRun = await actor.start();

import { ApifyClient } from "apify-client"
const apify = new ApifyClient({ token: 'my-token' });


// Resource clients accept an ID of the resource.
const actor = apify.actor('john-doe/my-actor');
// Fetches the john-doe/my-actor object from the API.
const myActor = await actor.get();
// Starts the run of john-doe/my-actor and returns the Run object.
const myActorRun = await actor.start();

The actor should ideally update my postgres database directly. All I need to do is set up a webhook receiver for incoming webhooks to get updated about what's happening.

Apify & Crawlee•3y ago•

1 reply

awake-maroon

How to call apify actors from a separate server?

import { ApifyClient } from "apify-client"
const apify = new ApifyClient({ token: 'my-token' });


// Resource clients accept an ID of the resource.
const actor = apify.actor('john-doe/my-actor');
// Fetches the john-doe/my-actor object from the API.
const myActor = await actor.get();
// Starts the run of john-doe/my-actor and returns the Run object.
const myActorRun = await actor.start();

import { ApifyClient } from "apify-client"
const apify = new ApifyClient({ token: 'my-token' });


// Resource clients accept an ID of the resource.
const actor = apify.actor('john-doe/my-actor');
// Fetches the john-doe/my-actor object from the API.
const myActor = await actor.get();
// Starts the run of john-doe/my-actor and returns the Run object.
const myActorRun = await actor.start();

The actor should ideally update my postgres database directly. All I need to do is set up a webhook receiver for incoming webhooks to get updated about what's happening.

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,851 Members

Join

How to call apify actors from a separate server?

How to call apify actors from a separate server?

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads