Proxy authentication bug?

Was trying to connect residential proxies to my crawler after using some other proxies and after changing ProxyConfiguration URLs I got error

net::ERR_TUNNEL_CONNECTION_FAILED

After checking credentials, and all possible whitelists and blacklists I realized that problem is not on my side. So I tried to implement proxy authentication in old way and it worked.

This code doesn't work:

import { PuppeteerCrawler, ProxyConfiguration } from 'crawlee';

import { router } from './routes.js';

const startUrls = ['https://crawlee.dev'];

const crawler = new PuppeteerCrawler({
  requestHandler: router,
  proxyConfiguration: new ProxyConfiguration({
    proxyUrls:   ['http://MyUserName:MyPassword@gate.smartproxy.com:7000'],
  }),
});

await crawler.run(startUrls);

import { PuppeteerCrawler, ProxyConfiguration } from 'crawlee';

import { router } from './routes.js';

const startUrls = ['https://crawlee.dev'];

const crawler = new PuppeteerCrawler({
  requestHandler: router,
  proxyConfiguration: new ProxyConfiguration({
    proxyUrls:   ['http://MyUserName:MyPassword@gate.smartproxy.com:7000'],
  }),
});

await crawler.run(startUrls);

import { PuppeteerCrawler, ProxyConfiguration } from 'crawlee';

import { router } from './routes.js';

const startUrls = ['https://crawlee.dev'];

const crawler = new PuppeteerCrawler({
  requestHandler: router,
  proxyConfiguration: new ProxyConfiguration({
    proxyUrls:   ['http://MyUserName:MyPassword@gate.smartproxy.com:7000'],
  }),
});

await crawler.run(startUrls);

import { PuppeteerCrawler, ProxyConfiguration } from 'crawlee';

import { router } from './routes.js';

const startUrls = ['https://crawlee.dev'];

const crawler = new PuppeteerCrawler({
  requestHandler: router,
  proxyConfiguration: new ProxyConfiguration({
    proxyUrls:   ['http://MyUserName:MyPassword@gate.smartproxy.com:7000'],
  }),
});

await crawler.run(startUrls);

This code works:

import { PuppeteerCrawler } from 'crawlee';
import { router } from './routes.js';

const startUrls = ['https://crawlee.dev'];

const crawler = new PuppeteerCrawler({
  requestHandler: router,
  launchContext: {
    launchOptions: {
      headless: false,
      args: ['--proxy-server=http://gate.smartproxy.com:7000'],
    },
  },
  preNavigationHooks: [
    async ({ request, page }, gotoOptions) => {
      await page.authenticate({ username: 'MyUserName', password: 'MyPassword' })
    },
  ],
});

await crawler.run(startUrls);

import { PuppeteerCrawler } from 'crawlee';
import { router } from './routes.js';

const startUrls = ['https://crawlee.dev'];

const crawler = new PuppeteerCrawler({
  requestHandler: router,
  launchContext: {
    launchOptions: {
      headless: false,
      args: ['--proxy-server=http://gate.smartproxy.com:7000'],
    },
  },
  preNavigationHooks: [
    async ({ request, page }, gotoOptions) => {
      await page.authenticate({ username: 'MyUserName', password: 'MyPassword' })
    },
  ],
});

await crawler.run(startUrls);

import { PuppeteerCrawler } from 'crawlee';
import { router } from './routes.js';

const startUrls = ['https://crawlee.dev'];

const crawler = new PuppeteerCrawler({
  requestHandler: router,
  launchContext: {
    launchOptions: {
      headless: false,
      args: ['--proxy-server=http://gate.smartproxy.com:7000'],
    },
  },
  preNavigationHooks: [
    async ({ request, page }, gotoOptions) => {
      await page.authenticate({ username: 'MyUserName', password: 'MyPassword' })
    },
  ],
});

await crawler.run(startUrls);

import { PuppeteerCrawler } from 'crawlee';
import { router } from './routes.js';

const startUrls = ['https://crawlee.dev'];

const crawler = new PuppeteerCrawler({
  requestHandler: router,
  launchContext: {
    launchOptions: {
      headless: false,
      args: ['--proxy-server=http://gate.smartproxy.com:7000'],
    },
  },
  preNavigationHooks: [
    async ({ request, page }, gotoOptions) => {
      await page.authenticate({ username: 'MyUserName', password: 'MyPassword' })
    },
  ],
});

await crawler.run(startUrls);

Should I create a bug report?

NeoNomade•8/21/23, 9:24 AM

nope it's not a bug.
there are proxy providers that require a different approach sometimes for example a session id.
I'm using a proxy provider that has the same behaviour, check their documentation for session id or for other ways to compose the url.

colossal-harlequinOP•8/21/23, 9:43 PM

@NeoNomade I used bright data and they also require basic authentication and It wasn't required to authenticate manually with .authenticate.authenticate I thought that crawlee handles it automatically with launch context depending on proxyUrl content. But I didn't find where exactly crawlee uses URLS to authenticate only validation and initialization.

MEE6•8/21/23, 9:43 PM

@5y just advanced to level 1! Thanks for your contributions!

NeoNomade•8/22/23, 2:29 PM

You don’t have to Authenticate manually.
You have to generate URLs with random session ids based on their logic. I have a function for that I’ll share in a few minutes, let me reach my computer