How does createSessionFunction create session when parallel requests are being made
How does createSessionFunction create session when parallel requests are being made
At a glance
The community member has a custom function that opens a browser to get cookies. The problem is that their machine is small, and when multiple sessions are created, it tries to open many browsers at the same time, which could overwhelm the system resources. The community member wants to make the creation of sessions sequential, so that only one browser instance is running at any point in time.
In the comments, a community member suggests using queues, such as bullmq, to manage the creation of sessions. This way, each createSession task would be queued, and the system resources would not be overwhelmed. Another community member suggests setting a low maxConcurrency value, which could be different for localhost and the Apify cloud.
There is no explicitly marked answer in the comments, but the suggestions provided by the community members could be a solution to the problem.
I have a custom function which open a browser to get cookies. The problem is my machine is very small and what would happen when multiple sessions are being made it will try to open many browsers at the same time. Can I somehow make the creation of sessions sequential ? so even though I need 1000s of sessions but at any point in time only one session is created and no session can be created in parallel. So only one browser instance will be running at any point in time.
createSessionFunction: async(sessionPool,options) => {
var new_session = new Session({ sessionPool });
var proxyurl = await proxyConfigurationDE.newUrl(new_session.id);
console.log(proxyurl, new_session.id);
var cookies_g = await getCookiesDE(proxyurl,'https://www.example.com');
console.log("cookies from playwright..",cookies_g);
new_session.setCookies(
cookies_g,
'https://www.example.com'
);
console.log("Checking cookies set..", new_session.getCookieString("example.com"))
return new_session;
}
