absent-sapphire•3y ago
How do we assign a session to a request without having to use proxy?
Can we include the session or the sessionId when running addRequest?
3 Replies
No, since Session and Request are 2 different things. You can manipulate the createSessionFunction or handle it in preNavigationHooks to set a session that you want. I would like to see more customizable SessionPool as well but you can work around it manually
stormy-gold•3y ago
@Lukas Krivka I have a similar problem. The session object I get in my requesthandler I use it to make a new crawler inside requesthandler to crawl the multiple other URLs from the same session. Because thats how a reaul user would do to craw 2-3 URLS in sequence. Is there anyother way of doing it ? The problem is even if the new crawler is initiated with the session used in the orignal crawler but still the crawler inside request handler is using my ip and not the proxy.
I'm not sure I get you here. There are plans to rework the session management in Crawlee, it is not very flexible now. Normally we would set https://crawlee.dev/api/core/interface/SessionOptions#maxUsageCount on the session to the rotation you want.