Dataset.open(..) doesn't init dataset - when called outside of handler

Hi

Due to performance issues - I want to move out from handler all possible awaitsawaits.
For example here:

router.addHandler('details', async ({request, page, enqueueLinks, log}) => {
  const data = await page.evaluate(() => {
    // collect data ..
    return collectedData;
  });
  const dataset = await Dataset.open('myData');
  await dataset.pushData(data);
})

router.addHandler('details', async ({request, page, enqueueLinks, log}) => {
  const data = await page.evaluate(() => {
    // collect data ..
    return collectedData;
  });
  const dataset = await Dataset.open('myData');
  await dataset.pushData(data);
})

I want to move out from handler - init of dataset - like:

const dataset = await Dataset.open('myData');

router.addHandler('details', async ({request, page, enqueueLinks, log}) => {
  const data = await page.evaluate(() => {
    // collect data ..
    return collectedData;
  });
  await dataset.pushData(data);
})

const dataset = await Dataset.open('myData');

router.addHandler('details', async ({request, page, enqueueLinks, log}) => {
  const data = await page.evaluate(() => {
    // collect data ..
    return collectedData;
  });
  await dataset.pushData(data);
})

but now dataset is not initialised on crawlee start.
Folder ./storage/datasets/myData./storage/datasets/myData is not created.
And I get log

WARN  PuppeteerCrawler: Reclaiming failed request back to the list or queue. Dataset with id: e4901ade-57c3-49ec-8300-5a96338d381b does not exist.

WARN  PuppeteerCrawler: Reclaiming failed request back to the list or queue. Dataset with id: e4901ade-57c3-49ec-8300-5a96338d381b does not exist.

How can I properly init dataset in this case?
Thank you
Cheers
GT

Apify & Crawlee•4y ago•

4 replies

wet-aqua

Dataset.open(..) doesn't init dataset - when called outside of handler

Hi

Due to performance issues - I want to move out from handler all possible awaitsawaits.
For example here:

router.addHandler('details', async ({request, page, enqueueLinks, log}) => {
  const data = await page.evaluate(() => {
    // collect data ..
    return collectedData;
  });
  const dataset = await Dataset.open('myData');
  await dataset.pushData(data);
})

router.addHandler('details', async ({request, page, enqueueLinks, log}) => {
  const data = await page.evaluate(() => {
    // collect data ..
    return collectedData;
  });
  const dataset = await Dataset.open('myData');
  await dataset.pushData(data);
})

I want to move out from handler - init of dataset - like:

const dataset = await Dataset.open('myData');

router.addHandler('details', async ({request, page, enqueueLinks, log}) => {
  const data = await page.evaluate(() => {
    // collect data ..
    return collectedData;
  });
  await dataset.pushData(data);
})

const dataset = await Dataset.open('myData');

router.addHandler('details', async ({request, page, enqueueLinks, log}) => {
  const data = await page.evaluate(() => {
    // collect data ..
    return collectedData;
  });
  await dataset.pushData(data);
})

but now dataset is not initialised on crawlee start.
Folder ./storage/datasets/myData./storage/datasets/myData is not created.
And I get log

WARN  PuppeteerCrawler: Reclaiming failed request back to the list or queue. Dataset with id: e4901ade-57c3-49ec-8300-5a96338d381b does not exist.

WARN  PuppeteerCrawler: Reclaiming failed request back to the list or queue. Dataset with id: e4901ade-57c3-49ec-8300-5a96338d381b does not exist.

How can I properly init dataset in this case?
Thank you
Cheers
GT

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,735 Members

Join

Dataset.open(..) doesn't init dataset - when called outside of handler

Dataset.open(..) doesn't init dataset - when called outside of handler

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads