Max redirects

At a glance

The community member is experiencing an error message related to a CheerioCrawler, where the request is being redirected 10 times and then aborted. A community member suggests that this usually means the enqueued URL is redirecting to a different domain, so the desired data would not be obtained. They mention that Crawlee automatically stops processing such URLs. The community member also suggests using the followRedirect option to control whether redirects should be followed or not.

Useful resources

LLouis Deconinck

I am getting this error message, how to best deal with it?
Reclaiming failed request back to the list or queue. Redirected 10 times. Aborting.
Can I increase the max number of redirects for my CheerioCrawler?

1 comment

OOleg V.

You don’t need to do anything with it. This usually means the enqueued URL redirects somewhere else (likely a different domain), so you wouldn’t get the desired data anyway. Crawlee automatically stops processing such URLs.

Also, just in case, there’s an option called followRedirect that might be helpful depending on your use case. It allows you to control whether redirects should be followed or not:
https://crawlee.dev/api/next/core/interface/HttpRequest#followRedirect

usage example:

Plain Text

const response = await gotScraping({
                url: 'https://www.example.com',
                method: 'POST',
                proxyUrl,
                body: `some_payload`,
                followRedirect: false
            }

Add a reply

Apify Discord Mirror

Max redirects