genetic-orange•2y ago
Pass the instance of AutoscaledPool to carwler
Hello !
Lets say I instantiate a new Cheerio Crawler and I want to have a AutoscaledPool, that I control in various placed, refferenced to my crawler. Exactly like we do with requestQueue: new RequestQueue.open... etc..
Is there any way to achieve that ? I want to have that pool in an external place, so I can call .pause() or resume as I wish.
Thanks!
3 Replies
I guess better to https://crawlee.dev/api/basic-crawler/interface/BasicCrawlerOptions#keepAlive and move logic to add requests to external source, "auto" scaling expected to be automatic 😉
genetic-orangeOP•2y ago
Ok, will look into that. Thanks !
@Ayar just advanced to level 1! Thanks for your contributions! 🎉