metropolitan-bronze•2y ago
What would be a recommended way of scraping content behind 2fa auth?
I want to scrape some of my personal data of a website that is behind a 2fa authentication, what would be the recommended way of getting around this? Basically I need to be able to interact with the UI each time a new session is created against the server.
4 Replies
Hi @Mikael Karon ,
I am sorry this question is out of the scope of Crawlee and it really depends on which kind of authentication are you using and how the security codes are delivered to you - SMS, authentication app and so on...
metropolitan-bronzeOP•2y ago
The basic question could be simplified as “can I introduce a manual step” in a crawl?
Running Crawlee based solution is just as any other node.js application - you may implement waiting for user's the prompt whenever you need, just make you will not run out of the timeouts while waiting for the input.
metropolitan-bronzeOP•2y ago
Hm. Ok. That could be a way. Basically do a separate node part that opens a browser and triggers 2fa, then collect what’s needed and add to the crawler session