Apify

Apify Discord Mirror

Updated 2 days ago

CheerioCrawler headerGenerator help

Crawlee
NNeoNomade | Scraping hellhound
·
Hello !

I kept reading the docs but couldn't find a clear information about this. When we use Puppeteer or Playwright we can tweak in browserPool the fingerprintGenerator. For Cheerio we have the headerGenerator from got, how we can adjust it inside the CheerioCrawler ?
L
f
2 comments
LLouis Deconinck
Here's an example on how to work with headerGeneratorOptions using the BasicCrawler. I would assume it works in the same way for the CheerioCrawler. https://crawlee.dev/docs/next/guides/got-scraping#useheadergenerator
ffoxt141
Hi! You could also attempt to add the following option to CheerioCrawler:

Plain Text
        preNavigationHooks: [
            async (crawlingContext, opts: OptionsInit) => {
                opts.headers = {
                    ...opts.headers,
                    // your headers
                };
            }
        ]
Add a reply
Sign up and join the conversation on Discord