Session Cookies - Apify & Crawlee

Drogbata · 2025-09-21T21:05:36.737Z

When i use bs4 crawler i would expect that session stores received cookies and then reuses in next requests but SessionCookies are empty. Any idea why and how to make it work?

M

Mantisus•9/22/25, 5:30 PM

Hey @Drogbata

SessionCookies

SessionCookies

SessionCookies

SessionCookies are tied to a specific

Session

Session

.

I think you are checking

SessionCookies

SessionCookies

SessionCookies

SessionCookies in another

Session

Session

when you check this.

You can either limit SessionPool to 1 session. Or store cookies in some external variable and pass cookies to each Session, for example, in pre_navigation_hook.

Documentation that may be useful to you
https://crawlee.dev/python/docs/guides/logging-in-with-a-crawler
https://crawlee.dev/python/docs/guides/session-management

D

DrogbataOP•9/23/25, 6:33 AM

@Mantisus I am using 1 session, my requests also are tied to specific session_id and doesnt work.

so pre_navigation_hook would be my solution but i had to ask because that seems like core feature and not something i have to workaround.

M

Mantisus•9/24/25, 4:54 AM

@Drogbata сan you provide an example of the code?

M

Mantisus•9/24/25, 6:55 AM

With the following test code, it works as expected.

async def test():
    crawler = BeautifulSoupCrawler()

    @crawler.router.default_handler
    async def set_cookies(ctx: BeautifulSoupCrawlingContext):
        ctx.log.info(f'Cookies set to {ctx.session.id}')

        await ctx.add_requests([Request.from_url('https://httpbin.org/cookies', label='GET', session_id=ctx.session.id)])

    @crawler.router.handler('GET')
    async def get_cookies(ctx: BeautifulSoupCrawlingContext):
        ctx.log.info(f'Cookies retrieved from {ctx.session.id}')
        print(ctx.session.cookies)
        print(ctx.http_response.read())

    await crawler.run(['https://httpbin.org/cookies/set/a/1'])

async def test():
    crawler = BeautifulSoupCrawler()

    @crawler.router.default_handler
    async def set_cookies(ctx: BeautifulSoupCrawlingContext):
        ctx.log.info(f'Cookies set to {ctx.session.id}')

        await ctx.add_requests([Request.from_url('https://httpbin.org/cookies', label='GET', session_id=ctx.session.id)])

    @crawler.router.handler('GET')
    async def get_cookies(ctx: BeautifulSoupCrawlingContext):
        ctx.log.info(f'Cookies retrieved from {ctx.session.id}')
        print(ctx.session.cookies)
        print(ctx.http_response.read())

    await crawler.run(['https://httpbin.org/cookies/set/a/1'])

async def test():
    crawler = BeautifulSoupCrawler()

    @crawler.router.default_handler
    async def set_cookies(ctx: BeautifulSoupCrawlingContext):
        ctx.log.info(f'Cookies set to {ctx.session.id}')

        await ctx.add_requests([Request.from_url('https://httpbin.org/cookies', label='GET', session_id=ctx.session.id)])

    @crawler.router.handler('GET')
    async def get_cookies(ctx: BeautifulSoupCrawlingContext):
        ctx.log.info(f'Cookies retrieved from {ctx.session.id}')
        print(ctx.session.cookies)
        print(ctx.http_response.read())

    await crawler.run(['https://httpbin.org/cookies/set/a/1'])

async def test():
    crawler = BeautifulSoupCrawler()

    @crawler.router.default_handler
    async def set_cookies(ctx: BeautifulSoupCrawlingContext):
        ctx.log.info(f'Cookies set to {ctx.session.id}')

        await ctx.add_requests([Request.from_url('https://httpbin.org/cookies', label='GET', session_id=ctx.session.id)])

    @crawler.router.handler('GET')
    async def get_cookies(ctx: BeautifulSoupCrawlingContext):
        ctx.log.info(f'Cookies retrieved from {ctx.session.id}')
        print(ctx.session.cookies)
        print(ctx.http_response.read())

    await crawler.run(['https://httpbin.org/cookies/set/a/1'])