Dynamic Proxies by Request?

Currently I set proxy on my crawler,

const proxyConfiguration = await Actor.createProxyConfiguration({groups: ['RESIDENTIAL', 'BUYPROXIES1234'], countryCode: 'US'});

const myCrawler = new PuppeteerCrawler({
    // To use the proxy IP session rotation logic, you must turn the proxy usage on.
    proxyConfiguration: proxyConfiguration,
    ......
});

const proxyConfiguration = await Actor.createProxyConfiguration({groups: ['RESIDENTIAL', 'BUYPROXIES1234'], countryCode: 'US'});

const myCrawler = new PuppeteerCrawler({
    // To use the proxy IP session rotation logic, you must turn the proxy usage on.
    proxyConfiguration: proxyConfiguration,
    ......
});

Then the crawler will use this proxyConfig for all requests.

Is there a way to intercept each request and change proxy dynamically? e.g. using this pseudo code,

crawler.on('request', (req) => {
  if (req.url === 'https://myweb.com') {
     req.proxy.disable();
     // or disable certain group
     req.proxy.disableIfGroup('RESIDENTIAL');
  }
  return req.continue();
});

crawler.on('request', (req) => {
  if (req.url === 'https://myweb.com') {
     req.proxy.disable();
     // or disable certain group
     req.proxy.disableIfGroup('RESIDENTIAL');
  }
  return req.continue();
});

Apify & Crawlee•4y ago•

4 replies

worthy-rose

Dynamic Proxies by Request?

Currently I set proxy on my crawler,

const proxyConfiguration = await Actor.createProxyConfiguration({groups: ['RESIDENTIAL', 'BUYPROXIES1234'], countryCode: 'US'});

const myCrawler = new PuppeteerCrawler({
    // To use the proxy IP session rotation logic, you must turn the proxy usage on.
    proxyConfiguration: proxyConfiguration,
    ......
});

const proxyConfiguration = await Actor.createProxyConfiguration({groups: ['RESIDENTIAL', 'BUYPROXIES1234'], countryCode: 'US'});

const myCrawler = new PuppeteerCrawler({
    // To use the proxy IP session rotation logic, you must turn the proxy usage on.
    proxyConfiguration: proxyConfiguration,
    ......
});

Then the crawler will use this proxyConfig for all requests.

Is there a way to intercept each request and change proxy dynamically? e.g. using this pseudo code,

crawler.on('request', (req) => {
  if (req.url === 'https://myweb.com') {
     req.proxy.disable();
     // or disable certain group
     req.proxy.disableIfGroup('RESIDENTIAL');
  }
  return req.continue();
});

crawler.on('request', (req) => {
  if (req.url === 'https://myweb.com') {
     req.proxy.disable();
     // or disable certain group
     req.proxy.disableIfGroup('RESIDENTIAL');
  }
  return req.continue();
});

Dynamic Proxies by Request?

Similar Threads

Dynamic Proxies by Request?

Similar Threads

Similar Threads

Similar Threads