correct-apricot•16mo ago

How do you extract data from DataSet.getData()

And where is this documented. I did not find any examples of it. Just storing it. How do I pull out a row of the data? And access the html returned during the scrape? I suggest updating simple crawler example with this. So we have crud not just create usage covered

1 Reply

Saurav Jain•16mo ago

Hey @dobrien 1. To extract data and retrieve data: open dataset:

Dataset.open(`dataset-name`)

Dataset.open(`dataset-name`)

get data from dataset:

const {items} = await dataset.getData();

const {items} = await dataset.getData();

2. How do i pull a row out of the data:

    const { items } = await dataset.getData({
        offset: rowNumber,
        limit: 1,
    });

    const { items } = await dataset.getData({
        offset: rowNumber,
        limit: 1,
    });

3. To access HTML during scrape: you need to first save HTML in the dataset and then access it using the 1 point. Documentation: https://crawlee.dev/api/core/class/Dataset#getData, https://crawlee.dev/api/core/interface/DatasetDataOptions alternatively, you can store HTML to KV store and add link to it to dataset similarly

DatasetDataOptions | API | Crawlee

Dataset | API | Crawlee

The Dataset class represents a store for structured data where each object stored has the same attributes, such as online store products or real estate offers. You can imagine it as a table, where each object is a row and its attributes are columns. Dataset is an append-only storage - you can only add new records to it but you cannot modify or...

How do you extract data from DataSet.getData()

Did you find this page helpful?