How to use MemoryStorage (mainly for RequestQueue) on t...

At a glance

The community member is running an actor that uses Cheerio, takes less than 20 minutes, and has around 1,000 requests. They want to optimize costs by using in-memory storage instead of the platform's RequestQueue. They tried setting storage: new MemoryStorage() but the actor still seems to use the platform's RequestQueue. Another community member suggests calling Configuration.getGlobalConfig().useStorageClient(new MemoryStorage()) after Apify.init() to use in-memory storage. The original poster later comments that they were able to cut the costs in half by using the in-memory storage, which will save them some money that they can spend on buying bikes.

Useful resources

sstrajk

My actor runs typically use Cheerio, take <20m and have around 1k requests.
For this scenario, costs for RequestQueue writes/reads are often higher than compute units. I wanted to experiment with using in-memory storage to optimize costs (i think I understand the associated risks and I'm ok with them).
I've tried setting storage: new MemoryStorage() in Actor.main second argument as noted in docs & TS definitions, but actor runs on platform still seems to use "platform RQ", not "in-memory one". Any pointers?

https://console.apify.com/actors/64sLcqgxq4IB5hZrI/runs/QIkqpBDa846Ftn5xK#storage

Attachment

3 comments

LLukas Krivka

Good question, asking the Crawlee gods

LLukas Krivka

So you should be able to do it if you call this after Apify.init()
Configuration.getGlobalConfig().useStorageClient(new MemoryStorage())

sstrajk

Thanks both to Discord god and Crawlee gods :yay:
I managed the cut the costs to half – it's just a cents, but multiplied with number of actors and schedule, it will some me some money I can spend on buying bikes 😄

Attachment

Add a reply

Apify Discord Mirror

How to use MemoryStorage (mainly for RequestQueue) on the platform?