Firefox, PlaywrightCrawler, SSL_ERROR_BAD_CERT

At a glance

The community member is encountering an SSL_ERROR_BAD_CERT_DOMAIN error when scraping a page with PlaywrightCrawler. The error occurs in the browser, not in the Crawlee/Playwright code. The community member wonders if they can set a "accept all certificates" flag in Firefox to resolve the issue.

In the comments, another community member suggests using the ignoreHTTPSErrors option in Playwright to address the issue. They provide an example of how to use this option in the PlaywrightCrawler code. However, the community member still needs to know how to actually add the ignoreHTTPSErrors flag.

Another comment mentions a fix proposed in a Discord channel, but does not provide any further details.

Useful resources

nnew_in_town

One of the pages I want to scrape with PlaywrightCrawler returns the SSL_ERROR_BAD_CERT_DOMAIN error.
I can reproduce this error when I open this URL in Firefox/Chrome - I see the browser shows the prompt with
the warning and asks "...do you want to proceed?"

So the error is from the browser, not from Crawlee/Playwright...

But... Firefox has so many flags/settings... may be I can somehow set the flag
"accept all certificates"?

2 comments

nnew_in_town

UPDATE:
I see ignoreHTTPSErrors in Playwright https://playwright.dev/docs/api/class-testoptions#test-options-ignore-https-errors

In the code like like this:

Plain Text

const crawler = new PlaywrightCrawler({

    autoscaledPoolOptions: {
    ...
    },


    headless: true,
    browserPoolOptions: {
        ...

        prePageCreateHooks: [
            (pageId, browserController, pageOptions) => {
               ...
            },
        ],
    },
    ...

    launchContext: {
            ...
        useIncognitoPages: true,
        launcher: myLauncher
    },

How to add the ignoreHTTPSErrors flag?

nnew_in_town

it seems... the fix proposed here:
https://discord.com/channels/801163717915574323/1064543660923965460/1064576558288883712

Add a reply

Apify Discord Mirror

Firefox, PlaywrightCrawler, SSL_ERROR_BAD_CERT_DOMAIN error