exotic-emerald•3y ago
How to use MemoryStorage (mainly for RequestQueue) on the platform?
My actor runs typically use Cheerio, take <20m and have around 1k requests.
For this scenario, costs for RequestQueue writes/reads are often higher than compute units. I wanted to experiment with using in-memory storage to optimize costs (i think I understand the associated risks and I'm ok with them).
I've tried setting
storage: new MemoryStorage() in Actor.main second argument as noted in docs & TS definitions, but actor runs on platform still seems to use "platform RQ", not "in-memory one". Any pointers?
https://console.apify.com/actors/64sLcqgxq4IB5hZrI/runs/QIkqpBDa846Ftn5xK#storage
2 Replies
Good question, asking the Crawlee gods
So you should be able to do it if you call this after
Apify.init()
Configuration.getGlobalConfig().useStorageClient(new MemoryStorage())exotic-emeraldOP•3y ago
Thanks both to Discord god @Lukas Krivka and Crawlee gods :yay:
I managed the cut the costs to half – it's just a cents, but multiplied with number of actors and schedule, it will some me some money I can spend on buying bikes 😄
