5 replies

Sessions and proxies?

I am having a hard time understanding sessions and proxies. I have the following crawler setup:

const crawler = new PuppeteerCrawler({
    requestList,
    useSessionPool: true,
    persistCookiesPerSession: true,
    proxyConfiguration,
    requestHandler: router,
    requestHandlerTimeoutSecs: 100,
    headless: false,
    minConcurrency: 20,
    maxConcurrency: 30,
    launchContext: {
        launcher: PuppeteerExtra,
        useIncognitoPages: true
    },
})

const crawler = new PuppeteerCrawler({
    requestList,
    useSessionPool: true,
    persistCookiesPerSession: true,
    proxyConfiguration,
    requestHandler: router,
    requestHandlerTimeoutSecs: 100,
    headless: false,
    minConcurrency: 20,
    maxConcurrency: 30,
    launchContext: {
        launcher: PuppeteerExtra,
        useIncognitoPages: true
    },
})

Basically I want to run the same task concurrently with different proxies. Unless I set

useIncognitoPages: true

useIncognitoPages: true

, only one session is used concurrently with one proxy. Is this how it should work? What is the point of having a session pool if only one is used?

Sessions and proxies?

Similar Threads