Crawlee+PlaywrightCrawler+proxy - original IP leaking t...

At a glance

The community member is running a program from a German datacenter that uses US residential proxies, but the original IP address is being detected by the website pixelscan.net. The community member is looking for a way to avoid this WebRTC leak. The discussion covers various proxy services, disabling WebRTC in Firefox, and potential solutions for headless Firefox. There is a mention of a pull request on the Apify fingerprint-suite repository that may address this issue, but it is not yet merged.

Useful resources

nnew_in_town

I'm running this simple program from a server in German datacenter with IP 167.235...
This program uses US residential proxies (rotating every 1min).

And I see that pixelscan.net is able to detect my original IP: 167.235...
On the attached screenshot you can find it under "WebRTC address"

So how to avoid this?

P.S:
and another problem I see - "Plugins Length", it is discussed here https://discord.com/channels/801163717915574323/1059483872271798333

Attachments

19 comments

LLeMoussel

What type of proxy are you using? Is it an 'Elite' proxy?

nnew_in_town

well, I do not know what an "Elite proxy" is :-)

i am using proxies from smartproxy.com
these are "residential", "rotating" proxies.
The US proxies availiable under "us.smartproxy.com:<port>" (they have proxies for almost every country and the global pool)

I think for $12.5 per GB, "pay as you go" - they are not bad.

Anyway, I am open to alternatives

P.S. lets discuss different proxy services here: https://discord.com/channels/801163717915574323/1060179502392684594

LLukas Krivka

Apify has the same price for residentials and they aggregate multiple providers, you can try on free plan. Will check the original issue

LLukas Krivka

Check the Web RTC leak

nnew_in_town

by the way, this: https://browserleaks.com/webrtc might be helpfull.

and, in the Firefox/Linux I use every day for browsing - all fields on that page are either "n/a" or "false" - and I do not even see that green START button (twicked my Firefox to prevent fingerprinting + "Privacy Badger" extension + "Canvas Blocker" extension)

I think all features related to WebRTC could be just disabled

nnew_in_town

To disable WebRTC:
media.peerconnection.enabled (WebRTC) = false

Taken from this article
https://restoreprivacy.com/firefox-privacy/

I think, it make sense to change configuration of headless Firefox. To set privacy.resistFingerprinting and some other flags described there.

I did it in my desktop browser - no WebRTC leak more.

So, who knows how to apply this to headless Firefox?

UPDATE:
https://stackoverflow.com/questions/53017490/how-can-i-disable-webrtc-local-ip-leak-with-puppeteer
looks like solution for Puppeteer...

ppetrpatek.

I believe that Firefox and Chrome have this setting disabled by default. I remember this was a huge security issue because you could access the IP addresses of other devices on the same network as the computer. I tested it on my browsers and have no leaks when using VPN or proxy. This might be because your proxy is sending x-forwarded headers. Let me know if you have a repro. I am curious to find out what is happening.

nnew_in_town

Firefox and Chrome have this setting disabled by default.

this is not true for Firefox! I changed the media.peerconnection.enabled according to above-mentioned article -- these settings was enabled by default.

This might be because your proxy is sending x-forwarded headers.

Who knows what the hell my proxy is doing...

- how to check for x-forwarded headers?
Just make request through my proxy to this address https://httpbin.org/get and look fo "x-forwarded" ?

ppetrpatek.

What is your Firefox version?

Yes the httpbin should work.

nnew_in_town

firefox used to run these tests:
/home/<my-user-name>/.cache/ms-playwright/firefox-1357/
(thats on a machine in datacenter)

firefox I use on desktop (to read this forum for example ) is: 108.02 (64bit)

nnew_in_town

this screenshot made in headless mode, httpbin.org/get was accessed through proxy.

and - the same program accessing pixelscan.net - it shows the WebRTC leak

Attachments

ppetrpatek.

This is interesting. I believe that this is connected to the DC proxy. However, you are right that this leak should be prevented in the crawlee. I will test it and process this change. Thanks a lot for bringing this up!

ppetrpatek.

So the only way how to do it programmatically (http://kb.mozillazine.org/User.js_file). It is necessary to create a user.js file and add it to your docker build flow.

nnew_in_town

the only way how to do it programmatically (http://kb.mozillazine.org/User.js_file).

Well, in the firefox I use for browsing every day - I have no WebRTC leak and I do not use additional User.js !

I just changed the media.peerconnection.enabled and few other settings, thats it.
Is it possible to pass this and few other settings to Firefox used by Playwright/Crawlee?
Some command-line flags?

nnew_in_town

...to create a user.js file and add it to your docker build flow.

I do not using Docker at all. I just installed Crawlee on a machine in datacener and thats it
I see that Firefox is here:
~/.cache/ms-playwright/firefox-1357/

nnew_in_town

Is there any news on this (WebRTC detection/leak) ?

PPepa J

Hi There is currently PR https://github.com/apify/fingerprint-suite/pull/141 for this.

nnew_in_town

Great news! So, in the next release this will be availiable, right?

PPepa J

It is not merged yet, so not sure it would be part of next release, if you need it now, you can pull the got branch and build and use it locally.

Add a reply

Apify Discord Mirror

Crawlee+PlaywrightCrawler+proxy - original IP leaking through WebRTC