relaxed-coral•3y ago
Proxy services - recommendations, feedback
For developers building scrapers/crawlers with Crawlee library - which proxy services you are using?
- Is it possible to use "US residential proxies" ?
- What do you think about quality of service?
- What about price?
14 Replies
I think they are generally very similar. You can try Apify which is wrapper around several providers.
- Buyproxies are cheap and reasonably Ok
- Oxylabs are high quality and expensive
- BrightData are classic, good quality, bit expensive
- NetNut is cheaper with reasonable quality
rival-black•3y ago
I'm using ProxyEmpire and I'm happy with the success rate.
PS: I'm affiliated with them.
US IP pool - yes
Service is fine. Uptime is good, support is also helpful
The price is fine, compared to others.
I tried to use Packetstream before, but that's a NO go for me anymore. They're "overcharging". I ran some test and if I spend 500 MB, for example, they'll count it as 800 MB
relaxed-coralOP•3y ago
@Lukas Krivka @ThePhantom Thanks for the info!
What is your experience with "Datacenter proxies"?
And smartproxy.com offers "Dedicated datacenter proxies" - what animal is this? have no idea
if I spend 500 MB, for example, they'll count it as 800 MBit is a good point! How do you measure these 500 MB?
rival-black•3y ago
There are different ways. One of the easier ones is Glasswire. It shows you how much bandwidth is per process on your PC.
Or you can do it programmatically, by calculating the size of the request you're sending and the response size that you're getting back for each request and summing them up.
Datacenter proxies are often used for scraping, since they tend to be cheaper. But if they're going to work is highly dependent on the website you're scraping. What I'm personally doing is trying with datacenter proxies first and if I'm not happy with the success rate, I'll move to residential/mobile ones.
relaxed-coralOP•3y ago
Glasswirewell, this is Windows *.exe and I am on Ubuntu...
programmatically, by calculating the sizeWho can share this code? Or, may be, there is some API somewhere in Crawlee? or in Playwright/in Firefox?
Yeah, we tested PacketStream and found this overcharging as well
We use datacenter proxies for 90%+ scrapers, the shared (many users can use the same IP) ones are good enough. Dedicated (only you can use it) have some usage for heavily scraped sites like Google or Amazon but usually are overkill.
You can also test vs some website and check in Chrome dev tool Network the size of the request. You are not able to check this in Node.js simply unfortunately since it doesn't give you access to compressed response (you would need to use some very low level TCP stuff I think)
relaxed-coralOP•3y ago
Indeed, it seems, for the most cases usage of Datacenter Proxies is just fine.
I did some research on prices, number of proxies and GB of traffic:
Looks like smartproxy is kind of optimal solution for cases when you need 100-200 proxies/IPs but you use very little traffic.
(this is my case: I cut off the images, CSS, and in some cases even JS from the sites I scrape. And I disable google/facebook analytics. After I implemented this - traffic dropped to almost ZERO - in most cases my scraper loads pure HTML, nothing else
see details here:
https://discord.com/channels/801163717915574323/1060986956961546320
https://discord.com/channels/801163717915574323/1067738438750711808
)
So, the question is:
Can somebody suggest a cheaper/better proxy provider than smartproxy? (for the case "100-200-300 IPs and little traffic")
UPDATE: well, there is even a market research paper about this stuff!
https://proxyway.com/research/market-research-2021-key-findings
Proxyway
Proxy Market Research 2021: Key Findings - Proxyway
We tested the 8 largest residential proxy providers and wrote a research about them. Here's a summary of the most important things we discovered.
can somebody suggest proxy that works for example on this page? https://www.casino.fr/ecommerce/affichageCatalog/WE80205/C-104051-pates
even fr residential proxy does not work (I am trying with my browser with switchy omega extension, so blocking is not because of automated browser)
relaxed-coralOP•2y ago
After almost a year... let us return to this "big proxy service price comparison" ))
What I can say about this stuff... At the moment smartproxy.com offers this: (I am not advertising it, just used this service some time... it works... and I am looking around, may be I can pay less somewhere?)
So the question is: can I have the same/similar service:
a combination of 1-3 GB traffic in "Residential proxies" and 30-100 GB traffic in "Datacenter proxies"
and pay the same amount or less?
you can try iproyal for residential proxies
relaxed-coralOP•2y ago
Thanks @HonzaS !
I see: iproyal.com/residential-proxies/
What is your experience with iproyal?
They have Datacenter proxies too, but it is... somehow... difficult to compare:
residentials works better than apify's residential at least for ticketmaster.co.uk
I did not try datacenter ones
@HonzaS just advanced to level 15! Thanks for your contributions! 🎉
conscious-sapphire•2y ago
My theory is that with Packetstream, they charge twice. Once for the transfer to them, then one for the transfer to you.