graceful-blue•3y ago

How to set cookie on Crawlee?

I want to set cookie on PlaywrightCrawler, but I can't find the tutorial on the documentation.

6 Replies

harsh-harlequin•3y ago

The browser controller has cookie manipulation functions - https://crawlee.dev/api/browser-pool/class/PlaywrightController

PlaywrightController | API | Crawlee

The BrowserController serves two purposes. First, it is the base class that specialized controllers like PuppeteerController or PlaywrightController extend. Second, it defines the public interface of the specialized classes which provide only private methods. Therefore, we do not keep documentation for the specialized classes, because it's...

harsh-harlequin•3y ago

You can also see preNavigationHooks [1] & postNavigationHooks [2] e.g.

    preNavigationHooks: [
      async (context) => {
        const url = context.request.url;
        const cookie = context.request.headers['cookie'];
        context.log.info(`pre-nav: cookie is '${cookie}'`);
      }],
    postNavigationHooks: [
      async (context) => {
        const url = context.request.url;
        const cookie = context.request.headers['cookie'];
        context.log.info(`post-nav: cookie is '${cookie}'`);
      }],

    preNavigationHooks: [
      async (context) => {
        const url = context.request.url;
        const cookie = context.request.headers['cookie'];
        context.log.info(`pre-nav: cookie is '${cookie}'`);
      }],
    postNavigationHooks: [
      async (context) => {
        const url = context.request.url;
        const cookie = context.request.headers['cookie'];
        context.log.info(`post-nav: cookie is '${cookie}'`);
      }],

[1] https://crawlee.dev/api/browser-crawler/interface/BrowserCrawlerOptions#preNavigationHooks [2]https://crawlee.dev/api/browser-crawler/interface/BrowserCrawlerOptions#postNavigationHooks

BrowserCrawlerOptions | API | Crawlee

graceful-blueOP•3y ago

can you give the code example? I still have no idea how to implement a PlaywrightController to set a cookie it's for getting the cookie, I want to get the cookie but I dont know how to set the cookie on preNavigatioNHooks and postNavigationHooks fixed using page.context() :)

MEE6•3y ago

@Bagas Wastu just advanced to level 1! Thanks for your contributions! 🎉

Lukas Krivka•3y ago

context.browserController has ton of functions for cookie manipulation https://crawlee.dev/api/browser-pool/class/PlaywrightController

PlaywrightController | API | Crawlee

firm-tan•3y ago

for future readers

const crawler = new PlaywrightCrawler({
    preNavigationHooks: [
        async (crawlingContext, gotoOptions) => {
            await crawlingContext.page.context().addCookies([{
                name: 'hasConsented',
                value: '1',
                domain: 'www.blabla.com',
                path: '/',
            }])
        },
    ],
})

const crawler = new PlaywrightCrawler({
    preNavigationHooks: [
        async (crawlingContext, gotoOptions) => {
            await crawlingContext.page.context().addCookies([{
                name: 'hasConsented',
                value: '1',
                domain: 'www.blabla.com',
                path: '/',
            }])
        },
    ],
})

How to set cookie on Crawlee?

Did you find this page helpful?