exotic-emerald
exotic-emerald•2y ago

How to rotate non-random sessions within a crawler?

How to make a crawler use non-random input based session ID while still making it possible to rotate/randomize and persist the session when needed (e.g. bad/blocked proxy)? E.g. first use session ID email and if the session is retired and request retried then use session ID email-randomNumber etc.
1 Reply
exotic-emerald
exotic-emeraldOP•2y ago
Note that this will hang the process: 
createSessionFunction: async sessionPool => await sessionPool.getSession(input.id) || new Session({id: input.id, sessionPool}),
createSessionFunction: async sessionPool => await sessionPool.getSession(input.id) || new Session({id: input.id, sessionPool}),
and this will create duplicit sessions: 
createSessionFunction: sessionPool => new Session({id: input.id, sessionPool}),
createSessionFunction: sessionPool => new Session({id: input.id, sessionPool}),
Let me simplify the question: How to supply a crawler with a custom session after checking it does not already exist?

Did you find this page helpful?