painful-plum•3y ago

Set 'ignoreHTTPSErrors' on a PlaywrightCrawler

Hi everyone, I need to set the ignoreHTTPSErrors flag for a Playwright crawler.

const crawler = new PlaywrightCrawler({
    launchContext: {
        launchOptions: {
            ignoreHTTPSErrors: true,
        }
    },
    requestHandler: router
});

seems to work, but errors on

Type '{ ignoreHTTPSErrors: true; }' is not assignable to type 'LaunchOptions'.\n Object literal may only specify known properties, and 'ignoreHTTPSErrors' does not exist in type 'LaunchOptions'.",

What's the standard way of setting this? Thanks in advance

10 Replies

MEE6•3y ago

@LeRiton just advanced to level 1! Thanks for your contributions! 🎉

Lukas Krivka•3y ago

The type is correct, this option is not there https://playwright.dev/docs/api/class-browsertype#browser-type-launch

BrowserType | Playwright

BrowserType provides methods to launch a specific browser instance or connect to an existing one. The following is a typical example of using Playwright to drive automation:

Lukas Krivka•3y ago

Can toy send me the site, I will test it

HonzaS•3y ago

it is here when you create new context https://playwright.dev/docs/api/class-browser#browser-new-context

painful-plumOP•3y ago

Hi Lukas, Honza, thanks for your answers! I'm not sure I understand, do you tell me that I need to create a new context to set this flag? I need this to be set at crawler level, can I define this flag for current browser context and those who will eventually be created? @HonzaS @Lukas Krivka can you help me with setting this flag at a crawler level? What am I doing wrong?

HonzaS•3y ago

I would try to change it in the browser context in preNavigationHooks but I am not sure it will work. If you share what site you want to scrape maybe more people can help you.

painful-plumOP•3y ago

I can't share any specific site because we wan't to provide a list of URL for this crawler, some of them in 'http'. This is why we wan't to set this flag at the crawler level.

relaxed-coral•3y ago

Could you please try adding

browserPoolOptions: {
            preLaunchHooks: [
                async (_, launchContext) => {
                    launchContext.ignoreHTTPSErrors = true;
                },
            ],
        },

browserPoolOptions: {
            preLaunchHooks: [
                async (_, launchContext) => {
                    launchContext.ignoreHTTPSErrors = true;
                },
            ],
        },

to crawler options? The thing is - launchContext has browser launch options, while ignoreHTTPSErrors is new context option, therefore you have to use the preLaunchHooks. I am not sure however whether it's launchContext.ignoreHTTPSErrors or launchContext.launchOptions.ignoreHTTPSErrors

painful-plumOP•3y ago

It works, thanks for this Andrey!

relaxed-coral•3y ago

Perfect, glad that it worked out in the end 👍

Set 'ignoreHTTPSErrors' on a PlaywrightCrawler

Did you find this page helpful?