How to "store" and "retrieve" a browser on a per user basis?

I am working on a Crawlee-based crawler that performs actions on a "per user" basis. Given this, i want to keep a configuration of the browser on a per user basis. I already store cookies, and push those into the browser before each page is loaded on behalf of the user. But, this is done in the same browser.. and I think this is throwing things off.

I run into problems where the cookies “go bad”. This could have NOTHING to do with the architecture. But, it seems to me that every browser is “different”, and as such I think that might be throwing an error.

Does anyone have any thoughts on how to store on a per user basis? We also configure the proxy to use a sticky proxy/IP if we can for each user as well.

Thanks!

brilliant-limeOP•4/12/24, 8:08 PM

Also, related to this, in our current/old system, we would open tabs for threaded execution. I'm not sure if I need todo something similar with Crawlee? And, given this, I'm not sure how this works.

primary-violet•4/12/24, 8:51 PM

For user profile isolation with the browser-based crawlers, you can use the launchContext.userDataDirlaunchContext.userDataDir option - this is basically a passthrough option for the Playwright / Puppeteer option of the same name (https://playwright.dev/docs/api/class-browsertype#browser-type-launch-persistent-context-option-user-data-dir).

const crawler = new PlaywrightCrawler({
    requestHandler: router,
    launchContext: {
        userDataDir: './user_data' // path to the folder where you want to store the per-user data.
    },
});

const crawler = new PlaywrightCrawler({
    requestHandler: router,
    launchContext: {
        userDataDir: './user_data' // path to the folder where you want to store the per-user data.
    },
});

primary-violet•4/12/24, 8:51 PM

Regarding the "threaded" execution, Crawlee handles per-request concurrency automatically, so you don't really have to care for it (it scales up and down based on the current system load).

brilliant-limeOP•4/12/24, 8:51 PM

Ok that's super helpful.

brilliant-limeOP•4/12/24, 8:52 PM

so, it basically works with playwright's options for this?

primary-violet•4/12/24, 8:55 PM

Yep, launchContext.userDataDirlaunchContext.userDataDir is just passed to Playwright afaik. You can pass more launch options to the browser (like CLI arguments) in launchContext.launchOptionslaunchContext.launchOptions (check out the TS type annotation in your IDE, it gives you all the options you can use)

MEE6•4/12/24, 8:55 PM

@vroomvroomvroom just advanced to level 1! Thanks for your contributions!

brilliant-limeOP•4/12/24, 8:55 PM

so, we're running in Kubernetes.. wiht multiple worker processes.. so i'd need to mount these standard data directory into all of my workers.. so it could get to the correct path.

brilliant-limeOP•4/12/24, 8:56 PM

we're already stuffing the browser before retrieve with cookies, and dumping them back after the page is loaded

brilliant-limeOP•4/12/24, 8:57 PM

this would be the data directory that would store "other" stuff.. i'd guess. that would help us keep things "clean" between users.

primary-violet•4/12/24, 9:00 PM

I don't think we ever tried anything like this, but yes - in theory, it should work like this

If you keep the mapping "one user = one userDataDir", you might even save yourself the hassle with injecting the cookies - the cookies are saved in the userDataDiruserDataDir (along with localStoragelocalStorage contents etc.) This also shows why you definitely shouldn't share the same userDataDiruserDataDir between multiple users

If you don't specify this option, Playwright generates a new ephemeral userDataDiruserDataDir for each script execution iirc.

brilliant-limeOP•4/12/24, 9:06 PM

brilliant-limeOP•4/12/24, 9:07 PM

thanks for that.. yeah, we have initial cookies we'd need to inject. but otherwise, yeah, that seems like it would be logical.