5 replies

Errors while attempting to use Apify Proxy

I have created a very simple actor that uses the

gotScraping

gotScraping

library to retrieve the HTML from a set of YouTube video pages. The page URLs are passed as an input array.

await Actor.init();

const input = await Actor.getInput<Inputs>();
const store = await Actor.openKeyValueStore('html-page-store');

for (const url of input!.urls) {
    log.info(`Fetching page HTML for video ID: ${url}`);

    try {
        const proxyConfiguration = await Actor.createProxyConfiguration({
            groups: ['RESIDENTIAL'],
            countryCode: 'US',
        });

        const proxyUrl = await proxyConfiguration!.newUrl();

        log.info(`Using proxy URL:`, { proxyUrl });

        const requestOptions = {
            url,
            proxyUrl,
            headers: { ...Constants.defaultRequestHeaders }
        };

        const response = await gotScraping.get(requestOptions);

        await store.setValue(url, response.body, {
            contentType: 'text/html',
        });

        log.info(`Data extracted for video ID: ${url}`);
    } catch (error) {
        log.error(`Error extracting data for video ID: ${url}`, { error });
    }
}

await Actor.exit();

await Actor.init();

const input = await Actor.getInput<Inputs>();
const store = await Actor.openKeyValueStore('html-page-store');

for (const url of input!.urls) {
    log.info(`Fetching page HTML for video ID: ${url}`);

    try {
        const proxyConfiguration = await Actor.createProxyConfiguration({
            groups: ['RESIDENTIAL'],
            countryCode: 'US',
        });

        const proxyUrl = await proxyConfiguration!.newUrl();

        log.info(`Using proxy URL:`, { proxyUrl });

        const requestOptions = {
            url,
            proxyUrl,
            headers: { ...Constants.defaultRequestHeaders }
        };

        const response = await gotScraping.get(requestOptions);

        await store.setValue(url, response.body, {
            contentType: 'text/html',
        });

        log.info(`Data extracted for video ID: ${url}`);
    } catch (error) {
        log.error(`Error extracting data for video ID: ${url}`, { error });
    }
}

await Actor.exit();

When the actor runs, it correctly retrieves the first page in the sequence, but crashes when attempting to retrieve a different proxy for the second page (see attached screenshot)

What puzzles me is that the error does not get caught by the

catch

catch

block in my own code; it seems to crash at a lower level.

I am also not clear why I am seeing the warning message

WARN  ProxyConfiguration: Apify Proxy access check timed out.

WARN  ProxyConfiguration: Apify Proxy access check timed out.

and how to correct this issue. None of the docs seem to mention it.

Lastly, can you please clarify: if I want a different proxy for every request, should my call to

Actor.createProxyConfiguration

Actor.createProxyConfiguration

be inside the loop, or prior to the loop?

Hoping somebody can advise. Many thanks,
Tim

Errors while attempting to use Apify Proxy

I have created a very simple actor that uses the

gotScraping

gotScraping

library to retrieve the HTML from a set of YouTube video pages. The page URLs are passed as an input array.

await Actor.init();

const input = await Actor.getInput<Inputs>();
const store = await Actor.openKeyValueStore('html-page-store');

for (const url of input!.urls) {
    log.info(`Fetching page HTML for video ID: ${url}`);

    try {
        const proxyConfiguration = await Actor.createProxyConfiguration({
            groups: ['RESIDENTIAL'],
            countryCode: 'US',
        });

        const proxyUrl = await proxyConfiguration!.newUrl();

        log.info(`Using proxy URL:`, { proxyUrl });

        const requestOptions = {
            url,
            proxyUrl,
            headers: { ...Constants.defaultRequestHeaders }
        };

        const response = await gotScraping.get(requestOptions);

        await store.setValue(url, response.body, {
            contentType: 'text/html',
        });

        log.info(`Data extracted for video ID: ${url}`);
    } catch (error) {
        log.error(`Error extracting data for video ID: ${url}`, { error });
    }
}

await Actor.exit();

await Actor.init();

const input = await Actor.getInput<Inputs>();
const store = await Actor.openKeyValueStore('html-page-store');

for (const url of input!.urls) {
    log.info(`Fetching page HTML for video ID: ${url}`);

    try {
        const proxyConfiguration = await Actor.createProxyConfiguration({
            groups: ['RESIDENTIAL'],
            countryCode: 'US',
        });

        const proxyUrl = await proxyConfiguration!.newUrl();

        log.info(`Using proxy URL:`, { proxyUrl });

        const requestOptions = {
            url,
            proxyUrl,
            headers: { ...Constants.defaultRequestHeaders }
        };

        const response = await gotScraping.get(requestOptions);

        await store.setValue(url, response.body, {
            contentType: 'text/html',
        });

        log.info(`Data extracted for video ID: ${url}`);
    } catch (error) {
        log.error(`Error extracting data for video ID: ${url}`, { error });
    }
}

await Actor.exit();

catch

catch

block in my own code; it seems to crash at a lower level.

I am also not clear why I am seeing the warning message

WARN  ProxyConfiguration: Apify Proxy access check timed out.

WARN  ProxyConfiguration: Apify Proxy access check timed out.

and how to correct this issue. None of the docs seem to mention it.

Lastly, can you please clarify: if I want a different proxy for every request, should my call to

Actor.createProxyConfiguration

Actor.createProxyConfiguration

be inside the loop, or prior to the loop?

Hoping somebody can advise. Many thanks,
Tim

Errors while attempting to use Apify Proxy

Errors while attempting to use Apify Proxy

Similar Threads

Similar Threads

Similar Threads