Use RabbitMQ as an alternative queue.
Hey guys, I tried a few ways, also read some experiments with the dialogue below. But the problem seems to me that Crawlee is quite slow at starting up or running pipelines.
After I add the URL (uniqueKey) to requestQueue. It takes about 2-3 minutes for it to launch or more links have to be added to the queue for it to start.
Can you guys help me?
crawlee-jsExternal request queue + external result storage, Crawlee as daemon process - how to implement it?
After I add the URL (uniqueKey) to requestQueue. It takes about 2-3 minutes for it to launch or more links have to be added to the queue for it to start.
Can you guys help me?
crawlee-jsExternal request queue + external result storage, Crawlee as daemon process - how to implement it?
message.txt5.76KB
