Wwet-aqua
querying data-set on filesystem - like SQL
Hi
Let's assume I scrap data to folders:
And I have sample products:
And sample categories:
How can I make queries on data-set - similary to SQL - for example 'join':
Guess it's not part of Crawlee - but maybe you have ideas how to query?
Thank you ;]
Let's assume I scrap data to folders:
./storage/datasets/
products
categories./storage/datasets/
products
categoriesAnd I have sample products:
{
"id": "product1",
"categoryId": "category1",
"name": "Hammer",
}
{
"id": "product2",
"categoryId": "category2",
"name": "Bread",
}
// ... 100 other products{
"id": "product1",
"categoryId": "category1",
"name": "Hammer",
}
{
"id": "product2",
"categoryId": "category2",
"name": "Bread",
}
// ... 100 other productsAnd sample categories:
{
"id": "category1",
"name": "Tools",
}
{
"id": "category2",
"name": "Food",
}{
"id": "category1",
"name": "Tools",
}
{
"id": "category2",
"name": "Food",
}How can I make queries on data-set - similary to SQL - for example 'join':
select p.* from product as p
join categories as c on p.categoryId = c.id
where c.name == 'Tools'select p.* from product as p
join categories as c on p.categoryId = c.id
where c.name == 'Tools'Guess it's not part of Crawlee - but maybe you have ideas how to query?
Thank you ;]
