Bbrilliant-lime
One-proxy, many-sessions?
Context: Proxy providers often provide a single
Hypothesis: When
Sorry for long winded question, it's my first time using Crawlee and I'm unsure how the details fit together. Thanks for any attention this may get.
proxyUrlproxyUrl from which any number of connections can be opened, i.e. each connection having a different IP despite proxyUrlproxyUrl all being seemingly the same. I wonder, is Crawlee able to create e.g. 100 SessionSessions (and rotate upon retireretire/markBadmarkBad), despite there only being one proxyUrlproxyUrl specified? From reading the docs, a round robin rotation mechanism is referred to and a sessionId-proxyUrl pair is mentioned, so I get the sense that Crawlee every one SessionSession locks exactly one proxyUrlproxyUrl, making redundant the setting sessionPoolOptions.maxPoolSize > 0sessionPoolOptions.maxPoolSize > 0 . In other words, each proxyConfiguration.proxyUrlproxyConfiguration.proxyUrl can have max. 1 session attached to it.Hypothesis: When
ProxyConfiguration.proxyUrls.length === 1ProxyConfiguration.proxyUrls.length === 1, even though e.g. sessionPoolOptions.maxPoolSize === 100sessionPoolOptions.maxPoolSize === 100, Crawlee can/will only create one SessionSession because Crawlee thinks there's only one IP available (or something). Does this happen?Sorry for long winded question, it's my first time using Crawlee and I'm unsure how the details fit together. Thanks for any attention this may get.
