Proxy fails on SSL secured(httpS) websites

At a glance

Hey!

I'm trying different proxy providers and I've noticed the issue in the title.

I'm setting the proxy in

Plain Text

proxyUrls

in the following format:

Plain Text

http://user:pass@host:port

as I usually do. But with the current providers I'm testing, the request will fail with either 407 (Proxy Authentication Required) or 422 responses.

Strangely enough, if tried with

Plain Text

curl -x 'proxy string from the same providers, in the same format' https://example.com

- it works.

Any idea what could be causing it?

10 comments

LLukas Krivka

Can you provide the target URL? The syntax looks correct

TThePhantom

example.com or amazon

LLukas Krivka

Can you share more of the code? No one ever reported this so it is unlikely it would be a bug in Crawlee

TThePhantom

Same proxy string works perfectly fine with curl, as mentioned above. Not sure why it "asks for authentication". Let me grab a screenshot with the resi proxies as well

Attachment

TThePhantom

There you go

Attachment

TThePhantom

Ahh, I think I know why it happens. There's a semicolon in the username and/or password in both cases. Let me see how should I escape it

AApifyBot

just advanced to level 2! Thanks for your contributions! 🎉

TThePhantom

Can't seem to do so, though. Should not need to escape semicolon as it's not really a special character, but Crawlee is not happy with it :\

And that's with Cheerio. Same formatting works just fine with Playwright

LLukas Krivka

I think I remember this issue but don't remember how I solved it 😄

TThePhantom

Haha, happens to me too. That's why I started to "write things down"(thanks note taking apps and Github gists).

Let me know if you remember the solution. It'll be helpful for others as well, I'm sure 🙂

Add a reply

Apify Discord Mirror

Proxy fails on SSL secured(httpS) websites