How do I remove Sec-Ch-Ua header generated by Chrome?
It's always the default HeadlessChrome value. Very easy to detect a bot despite Crawlee's fingerprinting
[crawlee.crawlers._playwright._playwright_crawler] INFO Crawling URL: https://www.httpbin.org/headers
[crawlee.crawlers._playwright._playwright_crawler] INFO Response headers: {
"headers": {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
"Accept-Encoding": "gzip, deflate, br, zstd",
"Accept-Language": "en-US",
"Host": "www.httpbin.org",
"Priority": "u=0, i",
"Sec-Ch-Ua": "\"HeadlessChrome\";v=\"143\", \"Chromium\";v=\"143\", \"Not A(Brand\";v=\"24\"",
"Sec-Ch-Ua-Mobile": "?0",
"Sec-Ch-Ua-Platform": "\"macOS\"",
"Sec-Fetch-Dest": "document",
"Sec-Fetch-Mode": "navigate",
"Sec-Fetch-Site": "none",
"Sec-Fetch-User": "?1",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.6 Safari/605.1.15",
"X-Amzn-Trace-Id": "Root=1-69495bc0-1b888c117c04745e46ba392f"
}
}[crawlee.crawlers._playwright._playwright_crawler] INFO Crawling URL: https://www.httpbin.org/headers
[crawlee.crawlers._playwright._playwright_crawler] INFO Response headers: {
"headers": {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
"Accept-Encoding": "gzip, deflate, br, zstd",
"Accept-Language": "en-US",
"Host": "www.httpbin.org",
"Priority": "u=0, i",
"Sec-Ch-Ua": "\"HeadlessChrome\";v=\"143\", \"Chromium\";v=\"143\", \"Not A(Brand\";v=\"24\"",
"Sec-Ch-Ua-Mobile": "?0",
"Sec-Ch-Ua-Platform": "\"macOS\"",
"Sec-Fetch-Dest": "document",
"Sec-Fetch-Mode": "navigate",
"Sec-Fetch-Site": "none",
"Sec-Fetch-User": "?1",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.6 Safari/605.1.15",
"X-Amzn-Trace-Id": "Root=1-69495bc0-1b888c117c04745e46ba392f"
}
}
