custom CheerioCrawler User-Agent?

hi, could anyone show me an example of CheerioCrawler User-Agent? I had tried preNavigationHookspreNavigationHooks but got a stranger error

  preNavigationHooks: [
    function customUserAgent(_ctx, opts = {}) {
      opts.headers = opts.headers || {};
      opts.useHeaderGenerator = true;
      opts.headerGeneratorOptions = {
        browsers: [ 'chrome', 'safari' ],
        devices: [ 'mobile' ],
        operatingSystems: [ 'android', 'ios' ],
      };
    },
  ]

  preNavigationHooks: [
    function customUserAgent(_ctx, opts = {}) {
      opts.headers = opts.headers || {};
      opts.useHeaderGenerator = true;
      opts.headerGeneratorOptions = {
        browsers: [ 'chrome', 'safari' ],
        devices: [ 'mobile' ],
        operatingSystems: [ 'android', 'ios' ],
      };
    },
  ]

but got error: RequestError: No headers based on this input can be generated.RequestError: No headers based on this input can be generated.

I had tried use this options with header-generatorheader-generator then it works, but don't work with CheerioCrawler hooks

Apify & Crawlee•4y ago•

7 replies

rubber-blue

custom CheerioCrawler User-Agent?

hi, could anyone show me an example of CheerioCrawler User-Agent? I had tried preNavigationHookspreNavigationHooks but got a stranger error

  preNavigationHooks: [
    function customUserAgent(_ctx, opts = {}) {
      opts.headers = opts.headers || {};
      opts.useHeaderGenerator = true;
      opts.headerGeneratorOptions = {
        browsers: [ 'chrome', 'safari' ],
        devices: [ 'mobile' ],
        operatingSystems: [ 'android', 'ios' ],
      };
    },
  ]

  preNavigationHooks: [
    function customUserAgent(_ctx, opts = {}) {
      opts.headers = opts.headers || {};
      opts.useHeaderGenerator = true;
      opts.headerGeneratorOptions = {
        browsers: [ 'chrome', 'safari' ],
        devices: [ 'mobile' ],
        operatingSystems: [ 'android', 'ios' ],
      };
    },
  ]

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

15,894 Members

Join

custom CheerioCrawler User-Agent?

custom CheerioCrawler User-Agent?

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads