conscious-sapphire•2y ago

How to inject storage state into crawler's page

How can a storageState be injected into crawler's page before the page is created? Basically, how to reproduce this but using storageState from the session instead:

        prePageCreateHooks: [
            async (pageId, browserController, pageOptions) => {
                const storageState = await KeyValueStore.getValue('storageState');
                if (pageOptions && storageState)
                    pageOptions.storageState = storageState;
            },
        ],

        prePageCreateHooks: [
            async (pageId, browserController, pageOptions) => {
                const storageState = await KeyValueStore.getValue('storageState');
                if (pageOptions && storageState)
                    pageOptions.storageState = storageState;
            },
        ],

6 Replies

optimistic-gold•2y ago

Hey @Tobey! Long time no see 🙂 Could you elaborate more on what exactly are you trying to achieve? To be frank - I am not really sure what's happening there :/

conscious-sapphireOP•2y ago

Hey @Andrey Bykov ! Indeed. I need to restore storageState from a session and don't know how to do that in a crawler... Ideally, I need access to browserPools context before it is created and before it calls newPage but haven't found a way. I want to pass options to the call to newPage which is the only way to inject the storageState in PW.

conscious-sapphireOP•2y ago

https://playwright.dev/docs/api/class-browsercontext#browser-context-storage-state

BrowserContext | Playwright

* extends: [EventEmitter]

flat-fuchsia•2y ago

Another apify alumni, bring beers 😄 I was trying to hijack newPage too recently, and found this old discussion mentioning that modifying finalPageOptions works only with useIncognitoPages: true And indeed, it works! (didn't have time to investigate why it is, but that's the world we live in 😄 ) https://discord.com/channels/801163717915574323/1018881795791532093/1019020911468359731

optimistic-gold•2y ago

@strajk and the thread you pointed brings another apify alumni 😄

conscious-sapphireOP•2y ago

Cheers! PW ignores the options passed from browserPool unless in incognito mode. Not sure why, looks like a bug to me... the code there is quite messy. I think there should be a way to intercept the browserPool calls to newContext() and inject the options before it's created but didn't find any. Do you know of any way?

How to inject storage state into crawler's page

Did you find this page helpful?