External Queue Provider
Hello, I started using Crawlee and am quite impressed with the speed and
anti-bot protection. However I do want to scale the system horizontally between different
machines (nodes) in cluster and therefore I need to have shared queue broker (like Redis or RabbitMQ).
Is it possible to configure Crawlee to use my own queue instead of local in-process one? Or There is no out-of-the-box
way and I should implement this myself listening to my Redis queue and adding to in-process queue myself?
anti-bot protection. However I do want to scale the system horizontally between different
machines (nodes) in cluster and therefore I need to have shared queue broker (like Redis or RabbitMQ).
Is it possible to configure Crawlee to use my own queue instead of local in-process one? Or There is no out-of-the-box
way and I should implement this myself listening to my Redis queue and adding to in-process queue myself?
