Can't purge named datasets

When I crate a named dataset like

const dataset = await Dataset.open("test");

const dataset = await Dataset.open("test");

const dataset = await Dataset.open("test");

const dataset = await Dataset.open("test"); and let the script run the data gets appended after each run. I tried to call

purgeDefaultStorages()

purgeDefaultStorages()

purgeDefaultStorages()

purgeDefaultStorages() but this has no effect. What am I doing wrong?

named dataset is not default storage, you need to https://crawlee.dev/api/core/class/Dataset#drop from your code

André Mácola•5/9/23, 10:23 PM

I solved this issue by changing the default folder itself per crawler.

André Mácola•5/9/23, 10:24 PM

something like

const spider: string = easy.opts('spider')
process.env.CRAWLEE_STORAGE_DIR = `./storage/${spider}

const spider: string = easy.opts('spider')
process.env.CRAWLEE_STORAGE_DIR = `./storage/${spider}