5 replies

Errors while attempting to use Apify Proxy

I have created a very simple actor that uses the gotScrapinggotScraping library to retrieve the HTML from a set of YouTube video pages. The page URLs are passed as an input array.

await Actor.init();

const input = await Actor.getInput<Inputs>();
const store = await Actor.openKeyValueStore('html-page-store');

for (const url of input!.urls) {
    log.info(`Fetching page HTML for video ID: ${url}`);

    try {
        const proxyConfiguration = await Actor.createProxyConfiguration({
            groups: ['RESIDENTIAL'],
            countryCode: 'US',
        });

        const proxyUrl = await proxyConfiguration!.newUrl();

        log.info(`Using proxy URL:`, { proxyUrl });

        const requestOptions = {
            url,
            proxyUrl,
            headers: { ...Constants.defaultRequestHeaders }
        };

        const response = await gotScraping.get(requestOptions);

        await store.setValue(url, response.body, {
            contentType: 'text/html',
        });

        log.info(`Data extracted for video ID: ${url}`);
    } catch (error) {
        log.error(`Error extracting data for video ID: ${url}`, { error });
    }
}

await Actor.exit();

await Actor.init();

const input = await Actor.getInput<Inputs>();
const store = await Actor.openKeyValueStore('html-page-store');

for (const url of input!.urls) {
    log.info(`Fetching page HTML for video ID: ${url}`);

    try {
        const proxyConfiguration = await Actor.createProxyConfiguration({
            groups: ['RESIDENTIAL'],
            countryCode: 'US',
        });

        const proxyUrl = await proxyConfiguration!.newUrl();

        log.info(`Using proxy URL:`, { proxyUrl });

        const requestOptions = {
            url,
            proxyUrl,
            headers: { ...Constants.defaultRequestHeaders }
        };

        const response = await gotScraping.get(requestOptions);

        await store.setValue(url, response.body, {
            contentType: 'text/html',
        });

        log.info(`Data extracted for video ID: ${url}`);
    } catch (error) {
        log.error(`Error extracting data for video ID: ${url}`, { error });
    }
}

await Actor.exit();

When the actor runs, it correctly retrieves the first page in the sequence, but crashes when attempting to retrieve a different proxy for the second page (see attached screenshot)

What puzzles me is that the error does not get caught by the catchcatch block in my own code; it seems to crash at a lower level.

I am also not clear why I am seeing the warning message WARN ProxyConfiguration: Apify Proxy access check timed out.WARN ProxyConfiguration: Apify Proxy access check timed out. and how to correct this issue. None of the docs seem to mention it.

Lastly, can you please clarify: if I want a different proxy for every request, should my call to Actor.createProxyConfigurationActor.createProxyConfiguration be inside the loop, or prior to the loop?

Hoping somebody can advise. Many thanks,
Tim

Apify & Crawlee•15mo ago•

5 replies

TimCoulter

Errors while attempting to use Apify Proxy

I have created a very simple actor that uses the gotScrapinggotScraping library to retrieve the HTML from a set of YouTube video pages. The page URLs are passed as an input array.

await Actor.init();

const input = await Actor.getInput<Inputs>();
const store = await Actor.openKeyValueStore('html-page-store');

for (const url of input!.urls) {
    log.info(`Fetching page HTML for video ID: ${url}`);

    try {
        const proxyConfiguration = await Actor.createProxyConfiguration({
            groups: ['RESIDENTIAL'],
            countryCode: 'US',
        });

        const proxyUrl = await proxyConfiguration!.newUrl();

        log.info(`Using proxy URL:`, { proxyUrl });

        const requestOptions = {
            url,
            proxyUrl,
            headers: { ...Constants.defaultRequestHeaders }
        };

        const response = await gotScraping.get(requestOptions);

        await store.setValue(url, response.body, {
            contentType: 'text/html',
        });

        log.info(`Data extracted for video ID: ${url}`);
    } catch (error) {
        log.error(`Error extracting data for video ID: ${url}`, { error });
    }
}

await Actor.exit();

await Actor.init();

const input = await Actor.getInput<Inputs>();
const store = await Actor.openKeyValueStore('html-page-store');

for (const url of input!.urls) {
    log.info(`Fetching page HTML for video ID: ${url}`);

    try {
        const proxyConfiguration = await Actor.createProxyConfiguration({
            groups: ['RESIDENTIAL'],
            countryCode: 'US',
        });

        const proxyUrl = await proxyConfiguration!.newUrl();

        log.info(`Using proxy URL:`, { proxyUrl });

        const requestOptions = {
            url,
            proxyUrl,
            headers: { ...Constants.defaultRequestHeaders }
        };

        const response = await gotScraping.get(requestOptions);

        await store.setValue(url, response.body, {
            contentType: 'text/html',
        });

        log.info(`Data extracted for video ID: ${url}`);
    } catch (error) {
        log.error(`Error extracting data for video ID: ${url}`, { error });
    }
}

await Actor.exit();

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

15,894 Members

Join

Errors while attempting to use Apify Proxy

Errors while attempting to use Apify Proxy

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads