Apify Discord Mirror

Updated 50 minutes ago

Redirect Control

Im trying to make a simple crawler, how do proper control the redirects? Some bad proxies sometimes redirect to auth page , in this case i want to mark the request as failed if the redirect URL ( target ) contains something like /auth/login. Whats the best to handle this scenarios and abort the request earlier?
A
n
3 comments
so each request is a session? say i send 3 urls to crawl would this mark them all as failed once the session is marked as bad? I think i might have explained myself incorrectly. This still lets the page navigate to the auth-login page, my question was if its possible to prevent a redirect on the main document and retire the session in case it is.
sessions defined by the session pool, so on blocking mark request session as "bad" to not continue with other requests if current one is blocked
Add a reply
Sign up and join the conversation on Discord