How can I add my own cookies to the crawler

Question

Hi, I'm using crawlee to fetch some data but I don't know how to add my own cookies in my crawler. I'm using Playwright to fetch cookies and after that I want to pass (in a session if it is possible) them to my BeautifulSoupCrawler.

Mantisus · Answer

HeyYou can pass cookies to Request headers.If you want to deal with cookies at the session level, now this approach varies depending on the client you are using. See - https://github.com/apify/crawlee-python/issues/933

frankman · Answer

Thanks @Mantisus . This was my soluction and it works for me:

Plain Text

        cookie_header = "; ".join([f"{cookie['name']}={cookie['value']}" for cookie in cookies])
        # type of cookies is a list of dicts
        headers = {
            'content-type': 'application/x-www-form-urlencoded',
            'accept-language': 'en-US,en;q=0.9',
            'cookie': cookie_header,
        }

        initial_req = Request.from_url(
                url=url,
                method="POST",
                label="LISTINGS",
                headers=headers,
                payload=urlencode(get_payload(0)).encode(),
                use_extended_unique_key=True,
                user_data={
                    "page_index": 0,
                    "headers": headers  
                },
            )
        
        await crawler.run([initial_req])

I use the headers in the next request (in the handler) that I have passed in user_data

ApifyBot · Answer

@frankman just advanced to level 2! Thanks for your contributions! 🎉

Apify Discord Mirror

How can I add my own cookies to the crawler