Proxy services - recommendations, feedback

At a glance

The post asks about proxy services used by developers building scrapers/crawlers with the Crawlee library, specifically regarding the use of US residential proxies, quality of service, and pricing. The comments provide various perspectives:

Community members suggest trying Apify, which is a wrapper around several providers, and mention Buyproxies, Oxylabs, BrightData, and NetNut as options with varying quality and pricing. One community member shares their experience with ProxyEmpire, stating they are happy with the success rate and service quality, but had issues with Packetstream's "overcharging".

The discussion also covers datacenter proxies, with community members noting they are often used for scraping as they tend to be cheaper, but the success rate can vary depending on the website being scraped. There is a detailed comparison of pricing and offerings from different providers, including Smartproxy, which is suggested as a potential optimal solution for cases where 100-200 proxies/IPs are needed but with low traffic usage.

The community members also discuss ways to measure the actual bandwidth usage, such as using tools like Glasswire or calculating the request and response sizes programmatically. There is no explicitly marked answer, but the discussion provides a good overview of the various proxy service options and considerations for developers building scra

Useful resources

nnew_in_town

For developers building scrapers/crawlers with Crawlee library - which proxy services you are using?

Is it possible to use "US residential proxies" ?
What do you think about quality of service?
What about price?

18 comments

LLukas Krivka

I think they are generally very similar. You can try Apify which is wrapper around several providers.

Buyproxies are cheap and reasonably Ok
Oxylabs are high quality and expensive
BrightData are classic, good quality, bit expensive
NetNut is cheaper with reasonable quality

TThePhantom

I'm using ProxyEmpire and I'm happy with the success rate.
PS: I'm affiliated with them.

US IP pool - yes

Service is fine. Uptime is good, support is also helpful

The price is fine, compared to others.

I tried to use Packetstream before, but that's a NO go for me anymore. They're "overcharging". I ran some test and if I spend 500 MB, for example, they'll count it as 800 MB

nnew_in_town

Thanks for the info!
What is your experience with "Datacenter proxies"?

And smartproxy.com offers "Dedicated datacenter proxies" - what animal is this? have no idea

nnew_in_town

if I spend 500 MB, for example, they'll count it as 800 MB

it is a good point!

How do you measure these 500 MB?

TThePhantom

There are different ways. One of the easier ones is Glasswire. It shows you how much bandwidth is per process on your PC.
Or you can do it programmatically, by calculating the size of the request you're sending and the response size that you're getting back for each request and summing them up.

TThePhantom

Datacenter proxies are often used for scraping, since they tend to be cheaper. But if they're going to work is highly dependent on the website you're scraping. What I'm personally doing is trying with datacenter proxies first and if I'm not happy with the success rate, I'll move to residential/mobile ones.

nnew_in_town

Glasswire

well, this is Windows *.exe and I am on Ubuntu...

programmatically, by calculating the size

Who can share this code? Or, may be, there is some API somewhere in Crawlee? or in Playwright/in Firefox?

LLukas Krivka

Yeah, we tested PacketStream and found this overcharging as well

LLukas Krivka

We use datacenter proxies for 90%+ scrapers, the shared (many users can use the same IP) ones are good enough. Dedicated (only you can use it) have some usage for heavily scraped sites like Google or Amazon but usually are overkill.

LLukas Krivka

You can also test vs some website and check in Chrome dev tool Network the size of the request. You are not able to check this in Node.js simply unfortunately since it doesn't give you access to compressed response (you would need to use some very low level TCP stuff I think)

nnew_in_town

Indeed, it seems, for the most cases usage of Datacenter Proxies is just fine.

I did some research on prices, number of proxies and GB of traffic:

Plain Text

$25/month, 40gb, proxyempire "Rotating Datacenter Proxies" https://proxyempire.io/rotating-datacenter-proxies/

$0.80/IP + $0.110/GB, brightdata datacenter-proxies https://brightdata.com/proxy-types/datacenter-proxies

$12.50/Month, 10 Semi-dedicated proxies https://buyproxies.org/panel/index.php?rp=/store/semi-dedicated-proxies

$50/month, 77GB, Datacenter Proxies https://oxylabs.io/products/datacenter-proxies/shared

$100/Month  100GB – NetNut https://netnut.io/datacenter-proxies/#datacenter-proxy-pricing

$10/month+VAT - 100 IPs, 50GB smartproxy.com https://smartproxy.com/proxies/datacenter-proxies/pricing
$19/month+VAT - 200 IPs, 50GB smartproxy.com https://smartproxy.com/proxies/datacenter-proxies/pricing

Looks like smartproxy is kind of optimal solution for cases when you need 100-200 proxies/IPs but you use very little traffic.

(this is my case: I cut off the images, CSS, and in some cases even JS from the sites I scrape. And I disable google/facebook analytics. After I implemented this - traffic dropped to almost ZERO - in most cases my scraper loads pure HTML, nothing else
see details here:
https://discord.com/channels/801163717915574323/1060986956961546320
https://discord.com/channels/801163717915574323/1067738438750711808
)

So, the question is:
Can somebody suggest a cheaper/better proxy provider than smartproxy? (for the case "100-200-300 IPs and little traffic")

UPDATE: well, there is even a market research paper about this stuff!
https://proxyway.com/research/market-research-2021-key-findings

HHonzaS

can somebody suggest proxy that works for example on this page? https://www.casino.fr/ecommerce/affichageCatalog/WE80205/C-104051-pates
even fr residential proxy does not work (I am trying with my browser with switchy omega extension, so blocking is not because of automated browser)

nnew_in_town

After almost a year... let us return to this "big proxy service price comparison" ))

What I can say about this stuff... At the moment smartproxy.com offers this: (I am not advertising it, just used this service some time... it works... and I am looking around, may be I can pay less somewhere?)

Plain Text

Residential proxies
   either "Pay As You Go" = $8.50 / 1GB
   or     2 GB / month    = $14 
   or     8 GB / month    = $52


Datacenter proxies
   100 IP (USA) + 50 GB traffic = $10
   100 IP (USA) + 100 GB traffic = $11

So the question is: can I have the same/similar service: a combination of 1-3 GB traffic in "Residential proxies" and 30-100 GB traffic in "Datacenter proxies"and pay the same amount or less?

HHonzaS

you can try iproyal for residential proxies

nnew_in_town

Thanks !
I see: iproyal.com/residential-proxies/

Plain Text

Residential proxies
  either $7     / 1GB
  or     $11.9  / 2GB

What is your experience with iproyal?

They have Datacenter proxies too, but it is... somehow... difficult to compare:

Plain Text

Datacenter proxies
  $1.56 per proxy /30 days
  $1.48 per proxy /60 days
  $1.39 per proxy /90 days

HHonzaS

residentials works better than apify's residential at least for ticketmaster.co.uk
I did not try datacenter ones

AApifyBot

just advanced to level 15! Thanks for your contributions! 🎉

wwflanagan

My theory is that with Packetstream, they charge twice. Once for the transfer to them, then one for the transfer to you.

Add a reply

Apify Discord Mirror

Proxy services - recommendations, feedback