Bug when using Apify.utils.requestAsBrowser: process seems to crash on timeout from the apify proxy

My client recently contacted me that their old apify actor does not work anymore. Upon changing the code to use the new node docker image and updating apify version a little bit everything seemed to resolve for me locally. But when on apify platform we started testing the actor with apify proxy turned on, from time to time we got the error attached in the screenshot. As I deduced, the issue seems to be coming from Apify.utils.requestAsBrowser -> http2-wrapper (source: https://github.com/szmarczak/http2-wrapper/issues/66). But I have got no idea on how to circumvent the error as we can not (?) simply use the inner libraries

Relevant code snippet:

response = await Apify.utils.requestAsBrowser({
                url: url,
                headers: {
                    'user-agent': session.userAgent,
                    'accept': 'application/json',
                    ...session.getHeaders()
                },
                proxyUrl: session.getProxyUrl()
            });

response = await Apify.utils.requestAsBrowser({
                url: url,
                headers: {
                    'user-agent': session.userAgent,
                    'accept': 'application/json',
                    ...session.getHeaders()
                },
                proxyUrl: session.getProxyUrl()
            });

GitHub

Issues · szmarczak/http2-wrapper

Use HTTP/2 the same way like HTTP/1. Contribute to szmarczak/http2-wrapper development by creating an account on GitHub.

Apify & Crawlee•4y ago•

4 replies

excited-coffee

Bug when using Apify.utils.requestAsBrowser: process seems to crash on timeout from the apify proxy

Relevant code snippet:

response = await Apify.utils.requestAsBrowser({
                url: url,
                headers: {
                    'user-agent': session.userAgent,
                    'accept': 'application/json',
                    ...session.getHeaders()
                },
                proxyUrl: session.getProxyUrl()
            });

response = await Apify.utils.requestAsBrowser({
                url: url,
                headers: {
                    'user-agent': session.userAgent,
                    'accept': 'application/json',
                    ...session.getHeaders()
                },
                proxyUrl: session.getProxyUrl()
            });