Apify Discord Mirror

Updated last week

More meaningful error than ERR_TUNNEL_CONNECTION_FAILED

Hi There.

I am using a proxy to crawl some sites and encounter a ERR_TUNNEL_CONNECTION_FAILED error.

I am using brightData as my proxy service. If i was to curl my proxy endpoint I get a meaningful errror. For example

HTTP/1.1 502 Could not resolve host demo-site.reckoniq.coms
x-brd-err-code: target_40001
x-brd-err-msg: Could not resolve host demo-site.reckoniq.coms
X-Luminati-Error: Could not resolve host demo-site.reckoniq.coms
x-brd-error: Could not resolve host demo-site.reckoniq.coms
Proxy-Connection: close

or

HTTP/1.1 403 Forbidden
x-brd-err-code: policy_20050
x-brd-err-msg: Forbidden: target site requires special permission. You are trying to access a target site which is not permitted by our compliance policy. In order to gain access you may need to undergo a KYC process, you can do so by filling in the form: https://brightdata.com/cp/kyc If you have already completed the KYC approval, please contact your account manager for further details.
X-Luminati-Error: Forbidden: target site requires special permission. Contact BrightData for assistance
x-brd-error: Forbidden: target site requires special permission. Contact BrightData for assistance
Proxy-Connection: close

I am wondering how i can surface those errors in crawlee rather than just getting

ERROR PlaywrightCrawler: Request failed and reached maximum retries. Error: Detected a session error, rotating session...
page.goto: net::ERR_TUNNEL_CONNECTION_FAILED at https://demo-site.reckoniq.coms/

There is no response in the error handler so things are not makinging it that far

Thanks for any help
L
t
2 comments
Hi @shansmithnz
I don't think we can get more from this error, it is what playwright gives us when it fails to establish the connection.
You can surface detailed proxy errors in Crawlee by modifying the error handling logic to capture and log specific proxy response codes like 502 or 403. To achieve this, ensure that your crawler checks for proxy-specific errors and logs the response messages, such as "Could not resolve host" or "Forbidden: target site requires special permission". Additionally, you may want to enhance your session management to handle proxy errors gracefully and provide meaningful feedback rather than a generic session error.
Add a reply
Sign up and join the conversation on Discord