Proxy authentication bug?

Was trying to connect residential proxies to my crawler after using some other proxies and after changing ProxyConfiguration URLs I got error

net::ERR_TUNNEL_CONNECTION_FAILED

After checking credentials, and all possible whitelists and blacklists I realized that problem is not on my side. So I tried to implement proxy authentication in old way and it worked.

This code doesn't work:

import { PuppeteerCrawler, ProxyConfiguration } from 'crawlee';

import { router } from './routes.js';

const startUrls = ['https://crawlee.dev'];

const crawler = new PuppeteerCrawler({
  requestHandler: router,
  proxyConfiguration: new ProxyConfiguration({
    proxyUrls:   ['http://MyUserName:MyPassword@gate.smartproxy.com:7000'],
  }),
});

await crawler.run(startUrls);

import { PuppeteerCrawler, ProxyConfiguration } from 'crawlee';

import { router } from './routes.js';

const startUrls = ['https://crawlee.dev'];

const crawler = new PuppeteerCrawler({
  requestHandler: router,
  proxyConfiguration: new ProxyConfiguration({
    proxyUrls:   ['http://MyUserName:MyPassword@gate.smartproxy.com:7000'],
  }),
});

await crawler.run(startUrls);

This code works:

import { PuppeteerCrawler } from 'crawlee';
import { router } from './routes.js';

const startUrls = ['https://crawlee.dev'];

const crawler = new PuppeteerCrawler({
  requestHandler: router,
  launchContext: {
    launchOptions: {
      headless: false,
      args: ['--proxy-server=http://gate.smartproxy.com:7000'],
    },
  },
  preNavigationHooks: [
    async ({ request, page }, gotoOptions) => {
      await page.authenticate({ username: 'MyUserName', password: 'MyPassword' })
    },
  ],
});

await crawler.run(startUrls);

import { PuppeteerCrawler } from 'crawlee';
import { router } from './routes.js';

const startUrls = ['https://crawlee.dev'];

const crawler = new PuppeteerCrawler({
  requestHandler: router,
  launchContext: {
    launchOptions: {
      headless: false,
      args: ['--proxy-server=http://gate.smartproxy.com:7000'],
    },
  },
  preNavigationHooks: [
    async ({ request, page }, gotoOptions) => {
      await page.authenticate({ username: 'MyUserName', password: 'MyPassword' })
    },
  ],
});

await crawler.run(startUrls);

Should I create a bug report?

Apify & Crawlee•3y ago•

4 replies

colossal-harlequin

Proxy authentication bug?

Was trying to connect residential proxies to my crawler after using some other proxies and after changing ProxyConfiguration URLs I got error

net::ERR_TUNNEL_CONNECTION_FAILED

import { PuppeteerCrawler, ProxyConfiguration } from 'crawlee';

import { router } from './routes.js';

const startUrls = ['https://crawlee.dev'];

const crawler = new PuppeteerCrawler({
  requestHandler: router,
  proxyConfiguration: new ProxyConfiguration({
    proxyUrls:   ['http://MyUserName:MyPassword@gate.smartproxy.com:7000'],
  }),
});

await crawler.run(startUrls);

import { PuppeteerCrawler, ProxyConfiguration } from 'crawlee';

import { router } from './routes.js';

const startUrls = ['https://crawlee.dev'];

const crawler = new PuppeteerCrawler({
  requestHandler: router,
  proxyConfiguration: new ProxyConfiguration({
    proxyUrls:   ['http://MyUserName:MyPassword@gate.smartproxy.com:7000'],
  }),
});

await crawler.run(startUrls);

This code works:

import { PuppeteerCrawler } from 'crawlee';
import { router } from './routes.js';

const startUrls = ['https://crawlee.dev'];

const crawler = new PuppeteerCrawler({
  requestHandler: router,
  launchContext: {
    launchOptions: {
      headless: false,
      args: ['--proxy-server=http://gate.smartproxy.com:7000'],
    },
  },
  preNavigationHooks: [
    async ({ request, page }, gotoOptions) => {
      await page.authenticate({ username: 'MyUserName', password: 'MyPassword' })
    },
  ],
});

await crawler.run(startUrls);

import { PuppeteerCrawler } from 'crawlee';
import { router } from './routes.js';

const startUrls = ['https://crawlee.dev'];

const crawler = new PuppeteerCrawler({
  requestHandler: router,
  launchContext: {
    launchOptions: {
      headless: false,
      args: ['--proxy-server=http://gate.smartproxy.com:7000'],
    },
  },
  preNavigationHooks: [
    async ({ request, page }, gotoOptions) => {
      await page.authenticate({ username: 'MyUserName', password: 'MyPassword' })
    },
  ],
});

await crawler.run(startUrls);

Should I create a bug report?

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,851 Members

Join

Proxy authentication bug?

Proxy authentication bug?

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads