deep-jade•16mo ago

Cheerio Fingerprint

Is there a way to use fingerprints with the Cheerio crawler? I need it to send Firefox headers. It's currently sending chromium ones

Host: localhost:8000
Connection: keep-alive
Content-Length: 0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 Edg/124.0.0.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Sec-Fetch-Site: same-site
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Ch-Ua: "Chromium";v="124", "Microsoft Edge";v="124", "Not-A.Brand";v="99"

Host: localhost:8000
Connection: keep-alive
Content-Length: 0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 Edg/124.0.0.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Sec-Fetch-Site: same-site
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Ch-Ua: "Chromium";v="124", "Microsoft Edge";v="124", "Not-A.Brand";v="99"

6 Replies

deep-jadeOP•16mo ago

or can i remove some headers?

adverse-sapphire•16mo ago

Hi @Tay, to use only Firefox headers, you can modify gotScraping headerGeneratorOptions in preNavigationHooks:

    ...
    preNavigationHooks: [
        (_, gotOptions) => {
            gotOptions.headerGeneratorOptions = {
                browsers: ['firefox'],
            };
        },
    ],
    ...

    ...
    preNavigationHooks: [
        (_, gotOptions) => {
            gotOptions.headerGeneratorOptions = {
                browsers: ['firefox'],
            };
        },
    ],
    ...

https://github.com/apify/got-scraping?tab=readme-ov-file#headergeneratoroptions

GitHub

GitHub - apify/got-scraping: HTTP client made for scraping based on...

HTTP client made for scraping based on got. Contribute to apify/got-scraping development by creating an account on GitHub.

deep-jadeOP•16mo ago

Thanks, is it possible to remove a header after generation?

MEE6•16mo ago

@Tay just advanced to level 1! Thanks for your contributions! 🎉

deep-jadeOP•16mo ago

i can add extra headers by appending to request.headers in preNavigationHooks but not remove it

Oleg V.•15mo ago

Did you try to make it undefined ? e.q. request.headers['my-header'] = undefined

Cheerio Fingerprint

Did you find this page helpful?