Error: PlaywrightCrawler:SessionPool:Session "Cookie no...

At a glance

The community member is using PlaywrightCrawler with Firefox and is encountering an error when accessing wellfound.com. The error message indicates that a cookie from a different subdomain (prod.website-files.com) is being set, which is causing issues. This is leading to HTTP 403 errors and captchas when navigating to other pages on the site. The community member has provided their code settings, including using a session pool and persisting cookies per session. A community member has responded, suggesting that the error makes sense as the browser's default behavior is to not set cookies for a different subdomain than the request.

nnew_in_town

I am using PlaywrightCrawler with Firefox. When accessing wellfound.com and see this error:

Plain Text

DEBUG PlaywrightCrawler:SessionPool:Session: Could not set cookies. {"errorMessages":["Cookie not in this host's domain. Cookie:prod.website-files.com Request:wellfound.com"]}

It might be that this cookie is something important: I'm navigating to another page on this site and get HTTP 403 and captcha...
How to fix this error?

Have these settings in code:

Plain Text

        
    useSessionPool: true,
    persistCookiesPerSession: true,
    sessionPoolOptions: {
         maxPoolSize: 300,
         sessionOptions:{
             maxAgeSecs: 70,
             maxUsageCount: 2,
         },
     },
     
    launchContext: {
        ...
        launchOptions: {
            bypassCSP: true,
            acceptDownloads: true,

1 comment

PPepa J

Hi @new_in_town As I understand the error it happens when the request is sent. It makes sense do not set cookie related to different subdomain than the request is - as this is also the browser default behavior.

Add a reply

Apify Discord Mirror

Error: PlaywrightCrawler:SessionPool:Session "Cookie not in this host's domain"