worthy-roseW
Apify & Crawlee4y ago
4 replies
worthy-rose

Dynamic Proxies by Request?

Currently I set proxy on my crawler,
const proxyConfiguration = await Actor.createProxyConfiguration({groups: ['RESIDENTIAL', 'BUYPROXIES1234'], countryCode: 'US'});

const myCrawler = new PuppeteerCrawler({
    // To use the proxy IP session rotation logic, you must turn the proxy usage on.
    proxyConfiguration: proxyConfiguration,
    ......
});

Then the crawler will use this proxyConfig for all requests.

Is there a way to intercept each request and change proxy dynamically? e.g. using this pseudo code,
crawler.on('request', (req) => {
  if (req.url === 'https://myweb.com') {
     req.proxy.disable();
     // or disable certain group
     req.proxy.disableIfGroup('RESIDENTIAL');
  }
  return req.continue();
});
Was this page helpful?