other-emerald•2y ago
How to run multiple threads in crawler with each thread using different session?
I have a scenario where site start throwing captcha is more than 3 threads are open in same session. I am using rotating proxies. However, when I try to run this in 10 threads, I get capctha immediately as all threads are in same session. How to use multiple session at the same time limiting each session number of threads ?
1 Reply
The best is probably to assign the sessions manually in preNavigationHooks and keep them in your state object