TimCoulterT
Apify & Crawleeโ€ข10mo agoโ€ข
5 replies
TimCoulter

Errors while attempting to use Apify Proxy

I have created a very simple actor that uses the
gotScraping
library to retrieve the HTML from a set of YouTube video pages. The page URLs are passed as an input array.

await Actor.init();

const input = await Actor.getInput<Inputs>();
const store = await Actor.openKeyValueStore('html-page-store');

for (const url of input!.urls) {
    log.info(`Fetching page HTML for video ID: ${url}`);

    try {
        const proxyConfiguration = await Actor.createProxyConfiguration({
            groups: ['RESIDENTIAL'],
            countryCode: 'US',
        });

        const proxyUrl = await proxyConfiguration!.newUrl();

        log.info(`Using proxy URL:`, { proxyUrl });

        const requestOptions = {
            url,
            proxyUrl,
            headers: { ...Constants.defaultRequestHeaders }
        };

        const response = await gotScraping.get(requestOptions);

        await store.setValue(url, response.body, {
            contentType: 'text/html',
        });

        log.info(`Data extracted for video ID: ${url}`);
    } catch (error) {
        log.error(`Error extracting data for video ID: ${url}`, { error });
    }
}

await Actor.exit();


When the actor runs, it correctly retrieves the first page in the sequence, but crashes when attempting to retrieve a different proxy for the second page (see attached screenshot)

What puzzles me is that the error does not get caught by the
catch
block in my own code; it seems to crash at a lower level.

I am also not clear why I am seeing the warning message
WARN  ProxyConfiguration: Apify Proxy access check timed out.
and how to correct this issue. None of the docs seem to mention it.

Lastly, can you please clarify: if I want a different proxy for every request, should my call to
Actor.createProxyConfiguration
be inside the loop, or prior to the loop?

Hoping somebody can advise. Many thanks,
Tim
image.png
Was this page helpful?