change storage dir programaticly
Hi All,
I want to use crawlee in kubernetes. I want my jobs to be resumable if a pod gets evicted so I have set up a PV for storage. This however poses an issue if I have multiple pods running at once. To solve this I want to change the storage dir programaticly when instanciating crawlee. I know I can do this through ENV vars however id prefer a more programatic solution if possible?
I have looked at the constructors for the (Playwright) Crawler and the Configuration class but I dont seem to be able to set it.
Thanks!
I want to use crawlee in kubernetes. I want my jobs to be resumable if a pod gets evicted so I have set up a PV for storage. This however poses an issue if I have multiple pods running at once. To solve this I want to change the storage dir programaticly when instanciating crawlee. I know I can do this through ENV vars however id prefer a more programatic solution if possible?
I have looked at the constructors for the (Playwright) Crawler and the Configuration class but I dont seem to be able to set it.
Thanks!
