Best practices for long living crawler & rabbitmq
Hi guys,
I’m here to ask best practices. I have coded a simple manager:
- starts crawlee instance and initializes queue object
- receives rabbitmq messages and pushes them to the queue
This is set up as keep alive crawler (it never quits, just awaits new messages). Thing is that it has some cache and some work around the queue and crawler to not actually let it close (for some reason it was just stopping and new messages were pushed tu queue but it wasn’t read by the crawler)
This made me wonder, maybe it should be built different?
Is there any resource that would help me learn about best practices in building such thing on crawlee? Docs lack long living crawler examples
I’ll add that my setup is using many different handlers for different sites - don’t know if it’s important for this question
I’m here to ask best practices. I have coded a simple manager:
- starts crawlee instance and initializes queue object
- receives rabbitmq messages and pushes them to the queue
This is set up as keep alive crawler (it never quits, just awaits new messages). Thing is that it has some cache and some work around the queue and crawler to not actually let it close (for some reason it was just stopping and new messages were pushed tu queue but it wasn’t read by the crawler)
This made me wonder, maybe it should be built different?
Is there any resource that would help me learn about best practices in building such thing on crawlee? Docs lack long living crawler examples
I’ll add that my setup is using many different handlers for different sites - don’t know if it’s important for this question
This is the official developer community of Apify and Crawlee.
13,739Members
Resources
Similar Threads
Was this page helpful?
