sacred-roseS
Apify & Crawlee•4y ago•
10 replies
sacred-rose

How to clear named KeyStores before every run?

I have this function
async function initialSetup() {
    // Clear previous data sets
    const storeKeys = ["categories_store", "details_store"]

    for (const storeKey of storeKeys) {
        try {
            const store = await KeyValueStore.open(storeKey)
            await store.drop()
        } catch (error) {
            console.log(error)
            continue
        }
    }

    // Create outputs directory if it doesn't exist
    if (!fs.existsSync("outputs")) {
        fs.mkdirSync("outputs")
    }
}

It is the first thing that runs before I start my crawler. It works as expected and drops both of the keystores.

But when I try to write fresh data to these stores again with this code
const categoriesStore = await KeyValueStore.open("categories_store")
await categoriesStore.setValue("categories", categories)

I get this error
INFO  PuppeteerCrawler: Starting the crawl
INFO  PuppeteerCrawler: enqueueing new URLs
Error: Key-value store with id: 6c47506c-2c01-4a6a-9eaa-567ee6f58e96 does not exist.
    at KeyValueStoreClient.throwOnNonExisting (C:\scrape_crawlee\node_modules\@crawlee\src\resource-clients\common\base-client.ts:11:15)
    at KeyValueStoreClient.setRecord (C:\scrape_crawlee\node_modules\@crawlee\src\resource-clients\key-value-store.ts:222:18)


I was expecting it to create a new store if didn't exist, but for some reason it doesn't and I am kinda lost with this error.

Any help would be appreciated!
Was this page helpful?