Lliving-lavender
retireBrowserAfterPageCount does not work with values above close to 30 for Playwright
Hi,
I am using Crawlee with Playwright. Some Pages require a login procedure, which works well by simply using page actions in Playwright. I am running into a problem though, because Crawlee opens a new browser after around every 30 pages, which deletes all Cookies/Local Storage and therefore fails the login. Relogging can be difficult or sometimes not technically possible.
I tried using the browserPoolOptions to prevent the launch of a new browser like this:
browserPoolOptions: {
retireBrowserAfterPageCount: 50,
},
Setting the value to 1 works well, but setting it to 50 or higher numbers does not have any effect. It still launches a new browser after around 30 pages.
Is this a known feature/bug? It would help greatly if I could run a crawl on one browser instance for several hundred pages, even if it is not recommended.
Thank you for your help!
I am using Crawlee with Playwright. Some Pages require a login procedure, which works well by simply using page actions in Playwright. I am running into a problem though, because Crawlee opens a new browser after around every 30 pages, which deletes all Cookies/Local Storage and therefore fails the login. Relogging can be difficult or sometimes not technically possible.
I tried using the browserPoolOptions to prevent the launch of a new browser like this:
browserPoolOptions: {
retireBrowserAfterPageCount: 50,
},
Setting the value to 1 works well, but setting it to 50 or higher numbers does not have any effect. It still launches a new browser after around 30 pages.
Is this a known feature/bug? It would help greatly if I could run a crawl on one browser instance for several hundred pages, even if it is not recommended.
Thank you for your help!
