recent-teal

Confusion around configuring Crawlee through a tor proxy

Here is the code I'm working with currently:

import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';
import { firefox } from 'playwright';


const startUrls = ['https://crawlee.dev'];
const BBCNewsOnionStartUrls = ['https://www.bbcnewsd73hkzno2ini43t4gblxvycyac5aw4gnv7t2rccijh7745uqd.onion/'];
const proxyConfiguration = new ProxyConfiguration({ proxyUrls: ['socks5://localhost:9050'] });

const crawler = new PlaywrightCrawler({
    launchContext: {
        launcher: firefox,
        launchOptions: {
            proxy: {
                server: 'socks5://localhost:9050'
            },
            headless: false,
        }
    },
    proxyConfiguration: proxyConfiguration,
    requestHandler: async ({ request, page, log }) => {
        const pageTitle = await page.title();
        log.info(`URL: ${request.loadedUrl} | Page title: ${pageTitle}`);
    },
    // Comment this option to scrape the full website.
    maxRequestsPerCrawl: 20,
    maxRequestsPerMinute: 10,
    maxConcurrency: 1,
    minConcurrency: 1,
    sameDomainDelaySecs: 1,
});

// await crawler.run(startUrls);
await crawler.run(BBCNewsOnionStartUrls);

import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';
import { firefox } from 'playwright';


const startUrls = ['https://crawlee.dev'];
const BBCNewsOnionStartUrls = ['https://www.bbcnewsd73hkzno2ini43t4gblxvycyac5aw4gnv7t2rccijh7745uqd.onion/'];
const proxyConfiguration = new ProxyConfiguration({ proxyUrls: ['socks5://localhost:9050'] });

const crawler = new PlaywrightCrawler({
    launchContext: {
        launcher: firefox,
        launchOptions: {
            proxy: {
                server: 'socks5://localhost:9050'
            },
            headless: false,
        }
    },
    proxyConfiguration: proxyConfiguration,
    requestHandler: async ({ request, page, log }) => {
        const pageTitle = await page.title();
        log.info(`URL: ${request.loadedUrl} | Page title: ${pageTitle}`);
    },
    // Comment this option to scrape the full website.
    maxRequestsPerCrawl: 20,
    maxRequestsPerMinute: 10,
    maxConcurrency: 1,
    minConcurrency: 1,
    sameDomainDelaySecs: 1,
});

// await crawler.run(startUrls);
await crawler.run(BBCNewsOnionStartUrls);

When I use the proxyConfiguration I run into the following error ERROR PlaywrightCrawler: Request failed and reached maximum retries. page.goto: NS_ERROR_UNKNOWN_PROXY_HOSTERROR PlaywrightCrawler: Request failed and reached maximum retries. page.goto: NS_ERROR_UNKNOWN_PROXY_HOST However, when I remove it, everything seems to work okay.

So my question is why isn't proxyConfigurationproxyConfiguration needed in this case? Are all requests still being directed through the tor proxy I have running locally?

Thanks!

(I have verified the tor proxy is running via curl --socks5-hostname localhost:9050 https://check.torproject.org/api/ipcurl --socks5-hostname localhost:9050 https://check.torproject.org/api/ip

Apify & Crawlee•2y ago

recent-teal

Confusion around configuring Crawlee through a tor proxy

Here is the code I'm working with currently:

import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';
import { firefox } from 'playwright';


const startUrls = ['https://crawlee.dev'];
const BBCNewsOnionStartUrls = ['https://www.bbcnewsd73hkzno2ini43t4gblxvycyac5aw4gnv7t2rccijh7745uqd.onion/'];
const proxyConfiguration = new ProxyConfiguration({ proxyUrls: ['socks5://localhost:9050'] });

const crawler = new PlaywrightCrawler({
    launchContext: {
        launcher: firefox,
        launchOptions: {
            proxy: {
                server: 'socks5://localhost:9050'
            },
            headless: false,
        }
    },
    proxyConfiguration: proxyConfiguration,
    requestHandler: async ({ request, page, log }) => {
        const pageTitle = await page.title();
        log.info(`URL: ${request.loadedUrl} | Page title: ${pageTitle}`);
    },
    // Comment this option to scrape the full website.
    maxRequestsPerCrawl: 20,
    maxRequestsPerMinute: 10,
    maxConcurrency: 1,
    minConcurrency: 1,
    sameDomainDelaySecs: 1,
});

// await crawler.run(startUrls);
await crawler.run(BBCNewsOnionStartUrls);

import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';
import { firefox } from 'playwright';


const startUrls = ['https://crawlee.dev'];
const BBCNewsOnionStartUrls = ['https://www.bbcnewsd73hkzno2ini43t4gblxvycyac5aw4gnv7t2rccijh7745uqd.onion/'];
const proxyConfiguration = new ProxyConfiguration({ proxyUrls: ['socks5://localhost:9050'] });

const crawler = new PlaywrightCrawler({
    launchContext: {
        launcher: firefox,
        launchOptions: {
            proxy: {
                server: 'socks5://localhost:9050'
            },
            headless: false,
        }
    },
    proxyConfiguration: proxyConfiguration,
    requestHandler: async ({ request, page, log }) => {
        const pageTitle = await page.title();
        log.info(`URL: ${request.loadedUrl} | Page title: ${pageTitle}`);
    },
    // Comment this option to scrape the full website.
    maxRequestsPerCrawl: 20,
    maxRequestsPerMinute: 10,
    maxConcurrency: 1,
    minConcurrency: 1,
    sameDomainDelaySecs: 1,
});

// await crawler.run(startUrls);
await crawler.run(BBCNewsOnionStartUrls);

No replies yet

Join the Discord to continue the conversation

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,735 Members

Join

Confusion around configuring Crawlee through a tor proxy

Confusion around configuring Crawlee through a tor proxy

No replies yet

Apify & Crawlee

No replies yet

Apify & Crawlee

Similar Threads