How to rotate non-random sessions within a crawler?

At a glance

The community member is asking how to make a crawler use a non-random input-based session ID while still being able to rotate/randomize and persist the session when needed (e.g., when a proxy is bad or blocked). The community members discuss two potential approaches, one of which will hang the process and the other will create duplicate sessions. The community member then simplifies the question to ask how to supply a crawler with a custom session after checking that it does not already exist. There is no explicitly marked answer.

TTobey

How to make a crawler use non-random input based session ID while still making it possible to rotate/randomize and persist the session when needed (e.g. bad/blocked proxy)? E.g. first use session ID email and if the session is retired and request retried then use session ID email-randomNumber etc.

3 comments

TTobey

Note that this will hang the process:

Plain Text

createSessionFunction: async sessionPool => await sessionPool.getSession(input.id) || new Session({id: input.id, sessionPool}),

TTobey

and this will create duplicit sessions:

Plain Text

createSessionFunction: sessionPool => new Session({id: input.id, sessionPool}),

TTobey

Let me simplify the question: How to supply a crawler with a custom session after checking it does not already exist?

Add a reply

Apify Discord Mirror

How to rotate non-random sessions within a crawler?