Oopposite-copper
Scraping auth-protected pages with CheerioCrawler, should I use Session?
I am trying to scrape some pages that only have certain information available when the user is logged in (as a personal project, I understand the risks)
At first, I tried to add a request to the queue that executes a POST request to perform a login, and then save those cookies into the route handler session using
However, for some reason the session is always empty (since the session was destroyed) and the next handler has always a new session, even though I set the following configuration to my crawler:
I've seen that
Am I understanding sessions wrong? Are they supposed to be only available when running Apify actors?
If so, what kind of flow should I use to include the authentication headers to all my requests?
Thank you in advance
PD: I want to run this scraper only in my local environment.
PD2: Basically what I would want to do is something similar to this
But, using CheerioCrawler and now imagine that the apify actor pages are auth-protected so you need login cookies, how would you do it then?
At first, I tried to add a request to the queue that executes a POST request to perform a login, and then save those cookies into the route handler session using
session.setCookiesFromResponseHowever, for some reason the session is always empty (since the session was destroyed) and the next handler has always a new session, even though I set the following configuration to my crawler:
I've seen that
session.isBlocked()session.isExpired()Am I understanding sessions wrong? Are they supposed to be only available when running Apify actors?
If so, what kind of flow should I use to include the authentication headers to all my requests?
Thank you in advance
PD: I want to run this scraper only in my local environment.
PD2: Basically what I would want to do is something similar to this
Apify Store scrapingBut, using CheerioCrawler and now imagine that the apify actor pages are auth-protected so you need login cookies, how would you do it then?
This is the official developer community of Apify and Crawlee.
14,091Members
Resources
Recent Announcements
Similar Threads
Was this page helpful?
