Apify & CrawleeA&CApify & Crawlee
Powered by
TimCoulterT
Apify & Crawlee•11mo ago•
5 replies
TimCoulter

Errors while attempting to use Apify Proxy

I have created a very simple actor that uses the
gotScraping
gotScraping
library to retrieve the HTML from a set of YouTube video pages. The page URLs are passed as an input array.

await Actor.init();

const input = await Actor.getInput<Inputs>();
const store = await Actor.openKeyValueStore('html-page-store');

for (const url of input!.urls) {
    log.info(`Fetching page HTML for video ID: ${url}`);

    try {
        const proxyConfiguration = await Actor.createProxyConfiguration({
            groups: ['RESIDENTIAL'],
            countryCode: 'US',
        });

        const proxyUrl = await proxyConfiguration!.newUrl();

        log.info(`Using proxy URL:`, { proxyUrl });

        const requestOptions = {
            url,
            proxyUrl,
            headers: { ...Constants.defaultRequestHeaders }
        };

        const response = await gotScraping.get(requestOptions);

        await store.setValue(url, response.body, {
            contentType: 'text/html',
        });

        log.info(`Data extracted for video ID: ${url}`);
    } catch (error) {
        log.error(`Error extracting data for video ID: ${url}`, { error });
    }
}

await Actor.exit();
await Actor.init();

const input = await Actor.getInput<Inputs>();
const store = await Actor.openKeyValueStore('html-page-store');

for (const url of input!.urls) {
    log.info(`Fetching page HTML for video ID: ${url}`);

    try {
        const proxyConfiguration = await Actor.createProxyConfiguration({
            groups: ['RESIDENTIAL'],
            countryCode: 'US',
        });

        const proxyUrl = await proxyConfiguration!.newUrl();

        log.info(`Using proxy URL:`, { proxyUrl });

        const requestOptions = {
            url,
            proxyUrl,
            headers: { ...Constants.defaultRequestHeaders }
        };

        const response = await gotScraping.get(requestOptions);

        await store.setValue(url, response.body, {
            contentType: 'text/html',
        });

        log.info(`Data extracted for video ID: ${url}`);
    } catch (error) {
        log.error(`Error extracting data for video ID: ${url}`, { error });
    }
}

await Actor.exit();


When the actor runs, it correctly retrieves the first page in the sequence, but crashes when attempting to retrieve a different proxy for the second page (see attached screenshot)

What puzzles me is that the error does not get caught by the
catch
catch
block in my own code; it seems to crash at a lower level.

I am also not clear why I am seeing the warning message
WARN  ProxyConfiguration: Apify Proxy access check timed out.
WARN  ProxyConfiguration: Apify Proxy access check timed out.
and how to correct this issue. None of the docs seem to mention it.

Lastly, can you please clarify: if I want a different proxy for every request, should my call to
Actor.createProxyConfiguration
Actor.createProxyConfiguration
be inside the loop, or prior to the loop?

Hoping somebody can advise. Many thanks,
Tim
image.png
Apify & Crawlee banner
Apify & CrawleeJoin
This is the official developer community of Apify and Crawlee.
13,923Members
Resources
Was this page helpful?

Similar Threads

Recent Announcements

Similar Threads

Using apify proxy
urgent-maroonUurgent-maroon / apify-platform
4y ago
Static Proxy Provider (Alternatives to Apify)
technical-ivoryTtechnical-ivory / apify-platform
3y ago
I want to use apify proxy configuration from local app.
spontaneous-moccasinSspontaneous-moccasin / apify-platform
3y ago
Required to use proxy?
Alexey UdovydchenkoAAlexey Udovydchenko / apify-platform
4y ago
Recent Announcements
ellativity

**Update to Store Publishing Terms and Acceptable Use Policy** Due to an influx of fraudulent reviews recently, Apify's Legal team has taken some actions to protect developers, customers, and Apify, by updating the Store Publishing Terms and Acceptable Use Policy. Please pay special attention to the updated terms in section 4 of the Store Publishing Terms here: https://docs.apify.com/legal/store-publishing-terms-and-conditions Additionally, please review the changes to section 2 of the Acceptable Use Policy here: https://docs.apify.com/legal/acceptable-use-policy If you have any questions, please ask them in <#1206131794261315594> so everyone can see the discussion. Thanks!

ellativity · 3w ago

ellativity

Hi @everyone I'm hanging out with the Creator team at Apify in https://discord.com/channels/801163717915574323/1430491198145167371 if you want to discuss Analytics and Insights!

ellativity · 4w ago

ellativity

2 things for <@&1092713625141137429> members today: 1. The Apify developer rewards program is open for registrations: https://apify.notion.site/developer-rewards This is the program where you will earn points for marketing activities. The rewards are still TBC, but the real purpose of the program is to help you structure your marketing activities and efforts. In the coming weeks, I will be populating that link with guides to help you identify the best ways to market your Actors, as well as scheduling workshops and office hours to help you create content and develop your own marketing strategy. 2. At 2PM CET (in about 80 minutes) there will be an office hour with the team behind Insights and Analytics, who want your feedback on how to improve analytics for you. Join us in https://discord.com/channels/801163717915574323/1430491198145167371 to share your ideas!

ellativity · 4w ago