rising-crimson•15mo ago
change storage dir programaticly
Hi All,
I want to use crawlee in kubernetes. I want my jobs to be resumable if a pod gets evicted so I have set up a PV for storage. This however poses an issue if I have multiple pods running at once. To solve this I want to change the storage dir programaticly when instanciating crawlee. I know I can do this through ENV vars however id prefer a more programatic solution if possible?
I have looked at the constructors for the (Playwright) Crawler and the Configuration class but I dont seem to be able to set it.
Thanks!
3 Replies
Hello,
There is https://crawlee.dev/api/core/interface/ConfigurationOptions, you probably have to pass a new StorageClient https://crawlee.dev/api/core/interface/StorageClient > https://crawlee.dev/api/memory-storage/interface/MemoryStorageOptions
StorageClient | API | Crawlee
Represents a storage capable of working with datasets, KV stores and request queues.
You can define
storageClientOptions in the Configuration constructor, where you can use the MemoryStorageOptions, see the links above. Then, you can pass the Configuration object to the crawler constructor.rising-crimsonOP•15mo ago
ah cool found it now, thanks for the help!