Alex I•4mo ago

How to use Apify Proxy with NodeJS Fetch API in an actor for non-browser data collection?

I’m building an Apify actor where I first populate userData with data from an external API (using fetch) and then make browser requests. How do I configure Apify Proxy for both stages to use the same proxies and ensure data consistency?

const options: RequestOptions = {
    url: url,
    userData: { apiResponce: apiResponce },
};
await crawler.run(options);

const options: RequestOptions = {
    url: url,
    userData: { apiResponce: apiResponce },
};
await crawler.run(options);

2 Replies

Hall•4mo ago

Someone will reply to you shortly. In the meantime, this might help: -# This post was marked as solved by Alex I. View answer.

Alex IOP•4mo ago

Huh, ok I follow to Ask AI and found https://docs.apify.com/sdk/js/docs/3.1/guides/apify-platform#apify-proxy

import { Actor } from 'apify';

// Create proxy configuration
const proxyConfiguration = await Actor.createProxyConfiguration();

// Get a proxy URL to use with fetch
const proxyUrl = await proxyConfiguration.newUrl();

// Use the same proxyConfiguration for both fetch and crawler
// First for your API calls
// Then pass it to your crawler configuration

import { Actor } from 'apify';

// Create proxy configuration
const proxyConfiguration = await Actor.createProxyConfiguration();

// Get a proxy URL to use with fetch
const proxyUrl = await proxyConfiguration.newUrl();

// Use the same proxyConfiguration for both fetch and crawler
// First for your API calls
// Then pass it to your crawler configuration

Apify Platform | SDK for JavaScript | Apify Documentation

Apify platform - large-scale and high-performance web scraping

How to use Apify Proxy with NodeJS Fetch API in an actor for non-browser data collection?

Did you find this page helpful?