Apify

Apify Discord Mirror

Updated 4 weeks ago

Issue with Residential Proxies

At a glance

The community member is experiencing an issue with using residential proxies, where they encounter an SSL certificate verification error. The community members suggest disabling certificate validation by using the HttpxHttpClient with verify=False. This solution is confirmed to work from version 0.5.0 onwards. Additionally, the community members note that if the HTTP requests are very slow, the timeout period for the handler may also need to be increased.

Useful resources
uuberpea5000
·
Hi there. Whenever I try to use residential proxies ('HTTP://groups-RESIDENTIAL:/...') I run into this error:

httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)

The 'auto' group seems to work fine. Can anyone tell me what I'm doing wrong here?

Thanks!
MMantisus
Marked as solution
Hey

Try disabling certificate validation

Plain Text
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee.http_clients import HttpxHttpClient


async def main() -> None:
    http_client = HttpxHttpClient(verify=False)
    crawler = BeautifulSoupCrawler(http_client=http_client)
View full solution
u
M
7 comments
uuberpea5000
Hmm. The automated response may have directed me to an answer: I'm crawling a .gov URL.
MMantisus
Hey

Try disabling certificate validation

Plain Text
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee.http_clients import HttpxHttpClient


async def main() -> None:
    http_client = HttpxHttpClient(verify=False)
    crawler = BeautifulSoupCrawler(http_client=http_client)
MMantisus
this work from version 0.5.0
uuberpea5000
Ah ha! Yes. That seems to do it. Much appreciated. (It alsoimplies the solution to my next problem: extending the httpx client timeout period.)
MMantisus
Note that if httpx will be very slow (more than a minute), you may also need to increase the timeout for the handler - https://crawlee.dev/python/api/class/BasicCrawlerOptions#request_handler_timeout
uuberpea5000
Thanks. Yes. Just had to do that as well 🙂
uuberpea5000
Got another question, but I'll start a new thread for it. Thanks again.
Add a reply
Sign up and join the conversation on Discord