brilliant-limeB
Apify & Crawlee•3y ago•
2 replies
brilliant-lime

One-proxy, many-sessions?

Context: Proxy providers often provide a single 
proxyUrl
 from which any number of connections can be opened, i.e. each connection having a different IP despite 
proxyUrl
 all being seemingly the same. I wonder, is Crawlee able to create e.g. 100 
Session
s (and rotate upon 
retire
/
markBad
), despite there only being one 
proxyUrl
 specified? From reading the docs, a round robin rotation mechanism is referred to and a sessionId-proxyUrl pair is mentioned, so I get the sense that Crawlee every one 
Session
 locks exactly one 
proxyUrl
, making redundant the setting 
sessionPoolOptions.maxPoolSize > 0
 . In other words, each 
proxyConfiguration.proxyUrl
 can have max. 1 session attached to it.

Hypothesis: When 
ProxyConfiguration.proxyUrls.length === 1
, even though e.g. 
sessionPoolOptions.maxPoolSize === 100
, Crawlee can/will only create one 
Session
 because Crawlee thinks there's only one IP available (or something). Does this happen?

Sorry for long winded question, it's my first time using Crawlee and I'm unsure how the details fit together. Thanks for any attention this may get.
Was this page helpful?