How to clear named KeyStores before every run?

I have this function

async function initialSetup() {
    // Clear previous data sets
    const storeKeys = ["categories_store", "details_store"]

    for (const storeKey of storeKeys) {
        try {
            const store = await KeyValueStore.open(storeKey)
            await store.drop()
        } catch (error) {
            console.log(error)
            continue
        }
    }

    // Create outputs directory if it doesn't exist
    if (!fs.existsSync("outputs")) {
        fs.mkdirSync("outputs")
    }
}

async function initialSetup() {
    // Clear previous data sets
    const storeKeys = ["categories_store", "details_store"]

    for (const storeKey of storeKeys) {
        try {
            const store = await KeyValueStore.open(storeKey)
            await store.drop()
        } catch (error) {
            console.log(error)
            continue
        }
    }

    // Create outputs directory if it doesn't exist
    if (!fs.existsSync("outputs")) {
        fs.mkdirSync("outputs")
    }
}

It is the first thing that runs before I start my crawler. It works as expected and drops both of the keystores.

But when I try to write fresh data to these stores again with this code

const categoriesStore = await KeyValueStore.open("categories_store")
await categoriesStore.setValue("categories", categories)

const categoriesStore = await KeyValueStore.open("categories_store")
await categoriesStore.setValue("categories", categories)

I get this error

INFO  PuppeteerCrawler: Starting the crawl
INFO  PuppeteerCrawler: enqueueing new URLs
Error: Key-value store with id: 6c47506c-2c01-4a6a-9eaa-567ee6f58e96 does not exist.
    at KeyValueStoreClient.throwOnNonExisting (C:\scrape_crawlee\node_modules\@crawlee\src\resource-clients\common\base-client.ts:11:15)
    at KeyValueStoreClient.setRecord (C:\scrape_crawlee\node_modules\@crawlee\src\resource-clients\key-value-store.ts:222:18)

INFO  PuppeteerCrawler: Starting the crawl
INFO  PuppeteerCrawler: enqueueing new URLs
Error: Key-value store with id: 6c47506c-2c01-4a6a-9eaa-567ee6f58e96 does not exist.
    at KeyValueStoreClient.throwOnNonExisting (C:\scrape_crawlee\node_modules\@crawlee\src\resource-clients\common\base-client.ts:11:15)
    at KeyValueStoreClient.setRecord (C:\scrape_crawlee\node_modules\@crawlee\src\resource-clients\key-value-store.ts:222:18)

I was expecting it to create a new store if didn't exist, but for some reason it doesn't and I am kinda lost with this error.

Any help would be appreciated!

Apify & Crawlee•4y ago•

10 replies

sacred-rose

How to clear named KeyStores before every run?

I have this function

async function initialSetup() {
    // Clear previous data sets
    const storeKeys = ["categories_store", "details_store"]

    for (const storeKey of storeKeys) {
        try {
            const store = await KeyValueStore.open(storeKey)
            await store.drop()
        } catch (error) {
            console.log(error)
            continue
        }
    }

    // Create outputs directory if it doesn't exist
    if (!fs.existsSync("outputs")) {
        fs.mkdirSync("outputs")
    }
}

async function initialSetup() {
    // Clear previous data sets
    const storeKeys = ["categories_store", "details_store"]

    for (const storeKey of storeKeys) {
        try {
            const store = await KeyValueStore.open(storeKey)
            await store.drop()
        } catch (error) {
            console.log(error)
            continue
        }
    }

    // Create outputs directory if it doesn't exist
    if (!fs.existsSync("outputs")) {
        fs.mkdirSync("outputs")
    }
}

It is the first thing that runs before I start my crawler. It works as expected and drops both of the keystores.

But when I try to write fresh data to these stores again with this code

const categoriesStore = await KeyValueStore.open("categories_store")
await categoriesStore.setValue("categories", categories)

const categoriesStore = await KeyValueStore.open("categories_store")
await categoriesStore.setValue("categories", categories)

I get this error

INFO  PuppeteerCrawler: Starting the crawl
INFO  PuppeteerCrawler: enqueueing new URLs
Error: Key-value store with id: 6c47506c-2c01-4a6a-9eaa-567ee6f58e96 does not exist.
    at KeyValueStoreClient.throwOnNonExisting (C:\scrape_crawlee\node_modules\@crawlee\src\resource-clients\common\base-client.ts:11:15)
    at KeyValueStoreClient.setRecord (C:\scrape_crawlee\node_modules\@crawlee\src\resource-clients\key-value-store.ts:222:18)

INFO  PuppeteerCrawler: Starting the crawl
INFO  PuppeteerCrawler: enqueueing new URLs
Error: Key-value store with id: 6c47506c-2c01-4a6a-9eaa-567ee6f58e96 does not exist.
    at KeyValueStoreClient.throwOnNonExisting (C:\scrape_crawlee\node_modules\@crawlee\src\resource-clients\common\base-client.ts:11:15)
    at KeyValueStoreClient.setRecord (C:\scrape_crawlee\node_modules\@crawlee\src\resource-clients\key-value-store.ts:222:18)

I was expecting it to create a new store if didn't exist, but for some reason it doesn't and I am kinda lost with this error.

Any help would be appreciated!

How to clear named KeyStores before every run?

Similar Threads

How to clear named KeyStores before every run?

Similar Threads

Similar Threads

Similar Threads