deep-jade•17mo ago

how to make follow_redirects=false in CheerioCrawler

How to disable url redirects in cheerio crawler

4 Replies

Oleg V.•17mo ago

Try to play around with PreNavigationHooks:

preNavigationHooks: [
    (crawlingContext, gotOptions) => {
        // ...
    },
]

preNavigationHooks: [
    (crawlingContext, gotOptions) => {
        // ...
    },
]

there must be the way via gotOptions https://crawlee.dev/api/next/cheerio-crawler/interface/CheerioCrawlerOptions#preNavigationHooks and got has needed option: https://github.com/sindresorhus/got/blob/main/documentation/2-options.md#followredirect

deep-jadeOP•17mo ago

@Oleg V. Could you please provide me an example on how i can call the got inside preNaviigationHooks beacuse i am not getting any suitable options that can be passed through gotOptions for redirect

Oleg V.•17mo ago

  
preNavigationHooks: [
      async ({ request }, gotOptions) => {
          const { url } = request;
          // do something with url if you need
 
          gotOptions.timeout = { // eaxmple of dealing with got options 
              request: 90_000,
              response: 120_000,
          };

          // try this:
          gotOptions.followRedirect = false;
      },
  ],

  
preNavigationHooks: [
      async ({ request }, gotOptions) => {
          const { url } = request;
          // do something with url if you need
 
          gotOptions.timeout = { // eaxmple of dealing with got options 
              request: 90_000,
              response: 120_000,
          };

          // try this:
          gotOptions.followRedirect = false;
      },
  ],

deep-jadeOP•17mo ago

Thanks @Oleg V. It worked. Appreciate it

how to make follow_redirects=false in CheerioCrawler

Did you find this page helpful?