national-gold•3y ago

Override browser permission on PuppeteerCrawler

Hi, a quick question: how do I override a certain permission on a page when using PuppeteerCrawler? Something like so:

...,
preNavigationHooks: [
  async (crawlingContext, gotoOptions) => {
    const { page, browserController: { browser } } = crawlingContext;

    const context = await browser.defaultBrowserContext();
    context.clearPermissionOverrides();
    context.overridePermissions('some-domain', ['some-feature']);
  }
],
...

...,
preNavigationHooks: [
  async (crawlingContext, gotoOptions) => {
    const { page, browserController: { browser } } = crawlingContext;

    const context = await browser.defaultBrowserContext();
    context.clearPermissionOverrides();
    context.overridePermissions('some-domain', ['some-feature']);
  }
],
...

When I run it, I get a warning message:

WARN  PuppeteerCrawler: Reclaiming failed request back to the list or queue. Cannot read private member from an object whose class did not declare it

I understand that I'm accessing private prop but how/where do I override such permissions on PuppeteerCrawler? 🙂 Thanks! 🙏

6 Replies

national-goldOP•3y ago

Hi @vladdy, can you please assist with this? 🙏

frail-apricot•3y ago

Oh..interesting Let me take a look

national-goldOP•3y ago

Thank you!

frail-apricot•3y ago

do you happen to have a stacktrace too? also, can you try getting the context via page.browserContext() instead (followed by the same context.overridePermissions calls)?

national-goldOP•3y ago

@stacktrace:

ERROR PuppeteerCrawler: Request failed and reached maximum retries. TypeError: Cannot read private member from an object whose class did not declare it
    at __classPrivateFieldGet (/path/node_modules/puppeteer/lib/cjs/puppeteer/common/Browser.js:25:94)
    at Proxy.defaultBrowserContext (/path/node_modules/puppeteer/lib/cjs/puppeteer/common/Browser.js:213:16)
    at PuppeteerCrawler.preNavigationHooks (file:///path/src/index.js:46:39)
    at PuppeteerCrawler._executeHooks (/path/node_modules/@crawlee/basic/internals/basic-crawler.js:794:23)
    at PuppeteerCrawler._handleNavigation (/path/node_modules/@crawlee/browser/internals/browser-crawler.js:263:20)
    at PuppeteerCrawler._runRequestHandler (/path/node_modules/@crawlee/browser/internals/browser-crawler.js:215:28)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async PuppeteerCrawler._runRequestHandler (/path/node_modules/@crawlee/puppeteer/internals/puppeteer-crawler.js:102:9)
    at async wrap (/path/node_modules/@apify/timeout/index.js:52:21)

ERROR PuppeteerCrawler: Request failed and reached maximum retries. TypeError: Cannot read private member from an object whose class did not declare it
    at __classPrivateFieldGet (/path/node_modules/puppeteer/lib/cjs/puppeteer/common/Browser.js:25:94)
    at Proxy.defaultBrowserContext (/path/node_modules/puppeteer/lib/cjs/puppeteer/common/Browser.js:213:16)
    at PuppeteerCrawler.preNavigationHooks (file:///path/src/index.js:46:39)
    at PuppeteerCrawler._executeHooks (/path/node_modules/@crawlee/basic/internals/basic-crawler.js:794:23)
    at PuppeteerCrawler._handleNavigation (/path/node_modules/@crawlee/browser/internals/browser-crawler.js:263:20)
    at PuppeteerCrawler._runRequestHandler (/path/node_modules/@crawlee/browser/internals/browser-crawler.js:215:28)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async PuppeteerCrawler._runRequestHandler (/path/node_modules/@crawlee/puppeteer/internals/puppeteer-crawler.js:102:9)
    at async wrap (/path/node_modules/@apify/timeout/index.js:52:21)

Cool, testing... Perfect! That works. Thanks! ✨

frail-apricot•3y ago

oh, sweet No worries

Override browser permission on PuppeteerCrawler

Did you find this page helpful?