Dynamically change dataset id based on root_domain

then-purple · 2025-04-15T13:43:54.371Z

Hey folks. I've attached an example of my code as a snippet Is it possible to dynamically change the dataset id so that each link has it's own dataset?

H

Hall•4/15/25, 1:43 PM

Someone will reply to you shortly. In the meantime, this might help:

M

Mantisus•4/15/25, 1:52 PM

Hey @Rykari

Note the Dataset class - https://crawlee.dev/python/api/class/Dataset

You can open different Datasets in handlers and write data to them

Dataset | API | Crawlee for Python · Fast, reliable Python web cra...

Crawlee helps you build and maintain your Python crawlers. It's open source and modern, with type hints for Python to help you catch bugs early.

T

then-purpleOP•4/15/25, 1:56 PM

Well, that was easy. Ty

T

then-purpleOP•4/15/25, 1:57 PM

Changed

            await context.push_data(data)

            await context.push_data(data)

            await context.push_data(data)

            await context.push_data(data)

To

            hostname = urlparse(context.request.url).hostname
            dataset = await Dataset.open(name=hostname)
            await dataset.push_data(data)

            hostname = urlparse(context.request.url).hostname
            dataset = await Dataset.open(name=hostname)
            await dataset.push_data(data)

            hostname = urlparse(context.request.url).hostname
            dataset = await Dataset.open(name=hostname)
            await dataset.push_data(data)

            hostname = urlparse(context.request.url).hostname
            dataset = await Dataset.open(name=hostname)
            await dataset.push_data(data)