Ccomprehensive-tomato
Crawlee Session Management use cookie from db
Hi, I'm new in here, and I'm a bit confused on how to use PuppeteerCrawlee + Session Management +Proxy Management with my own cookies.
I got a bunch of instagram cookies in my db, and I want to pass it to session management / session pool.
My scraping operation can't be done without authenticated cookies, so I need to inject the cookies from the start.
I've already read the docs and playing the SessionPool, but it seems the default behavior of SessionPool is to use the cookies from the puppeteer itself.
Does that mean I need to set the cookies straight in the requestHandler everytime? but if I do that, how about the automatic pair matching between session + proxy? I've read in the docs that the Crawlee automatically pick the best pair of proxy + session to avoid ban.
And how exactly this automatic pair works? Instagram is very strict with the anti-bot countermeasure, so I just need to know how it really works behind the scene, and decide if it really suitable for my usecase.
Thank you
I got a bunch of instagram cookies in my db, and I want to pass it to session management / session pool.
My scraping operation can't be done without authenticated cookies, so I need to inject the cookies from the start.
I've already read the docs and playing the SessionPool, but it seems the default behavior of SessionPool is to use the cookies from the puppeteer itself.
Does that mean I need to set the cookies straight in the requestHandler everytime? but if I do that, how about the automatic pair matching between session + proxy? I've read in the docs that the Crawlee automatically pick the best pair of proxy + session to avoid ban.
And how exactly this automatic pair works? Instagram is very strict with the anti-bot countermeasure, so I just need to know how it really works behind the scene, and decide if it really suitable for my usecase.
Thank you
