MatzeM
Apify & Crawlee•2y ago•
4 replies
Matze

Bind session and proxy together

Hi,
I have a small problem, my sessions and proxies don't stay together, which I expected to be the default.

new PlaywrightCrawler({
 ...
  useSessionPool: true,
  sessionPoolOptions: {
    blockedStatusCodes: [403],
    sessionOptions: {
      maxErrorScore: 1,
      maxUsageCount: 7
    }
  },
  proxyConfiguration: new ProxyConfiguration({ proxyUrls: proxyList() }), // List of 250 Proxies
...
})

When I log the session id and port from the list in my router, the proxy.sessionId does not match the session.id.
 log.info(session?.id)
 log.info(proxyInfo?.port)
 log.info(proxyInfo?.sessionId)


Results look like:
INFO  PlaywrightCrawler: session_AlZoomLhQU
INFO  PlaywrightCrawler: 10209
INFO  PlaywrightCrawler: session_Dnha2MhDeX
....
INFO  PlaywrightCrawler: session_AlZoomLhQU
INFO  PlaywrightCrawler: 10208
INFO  PlaywrightCrawler: session_6jOviCJSHt
...


I don't know if the session may change afterwards after the proxy is assigned:
* https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L504
* https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L534
Was this page helpful?