Handle a 401 in errorHandler by detecting login form and gracefully continuing if present
Hello there!
I'm working on a page crawler that can handle logging into sites, and then crawling around as that user. We've had a lot of success so far with Crawlee (PuppeteerCrawler) by detecting the login in
Recently we were asked to support "logging in" to a simple password protection screen on a Netlify site.
On navigation to the page, the page returns a 401 status code but renders the password login form. Because of the 401 status code, Crawlee sees that and calls the
I can enqueue links from the page but the next request it tries to load, it gets the 401 error again. I'm guessing a little bit but I think the page is closed at the end of the
Is there anything I can do to abort the error handling flow from
I attempted to add a code example but hit the message limit. Can try in a follow up comment.
I'm working on a page crawler that can handle logging into sites, and then crawling around as that user. We've had a lot of success so far with Crawlee (PuppeteerCrawler) by detecting the login in
requestHandlerRecently we were asked to support "logging in" to a simple password protection screen on a Netlify site.
On navigation to the page, the page returns a 401 status code but renders the password login form. Because of the 401 status code, Crawlee sees that and calls the
errorHandlerI can enqueue links from the page but the next request it tries to load, it gets the 401 error again. I'm guessing a little bit but I think the page is closed at the end of the
errorHandlerIs there anything I can do to abort the error handling flow from
errorHandlerI attempted to add a code example but hit the message limit. Can try in a follow up comment.
