Oopposite-copper
Scraping auth-protected pages with CheerioCrawler, should I use Session?
I am trying to scrape some pages that only have certain information available when the user is logged in (as a personal project, I understand the risks)
At first, I tried to add a request to the queue that executes a POST request to perform a login, and then save those cookies into the route handler session using
However, for some reason the session is always empty (since the session was destroyed) and the next handler has always a new session, even though I set the following configuration to my crawler:
I've seen that
Am I understanding sessions wrong? Are they supposed to be only available when running Apify actors?
If so, what kind of flow should I use to include the authentication headers to all my requests?
Thank you in advance
PD: I want to run this scraper only in my local environment.
PD2: Basically what I would want to do is something similar to this
But, using CheerioCrawler and now imagine that the apify actor pages are auth-protected so you need login cookies, how would you do it then?
At first, I tried to add a request to the queue that executes a POST request to perform a login, and then save those cookies into the route handler session using
session.setCookiesFromResponsesession.setCookiesFromResponse, and afterwards add the starting point for my scraping.However, for some reason the session is always empty (since the session was destroyed) and the next handler has always a new session, even though I set the following configuration to my crawler:
I've seen that
session.isBlocked()session.isBlocked() and session.isExpired()session.isExpired() are always true, even before I set the cookies from the login response.Am I understanding sessions wrong? Are they supposed to be only available when running Apify actors?
If so, what kind of flow should I use to include the authentication headers to all my requests?
Thank you in advance
PD: I want to run this scraper only in my local environment.
PD2: Basically what I would want to do is something similar to this
Apify Store scrapingApify Store scraping https://crawlee.dev/docs/introduction/scrapingBut, using CheerioCrawler and now imagine that the apify actor pages are auth-protected so you need login cookies, how would you do it then?
