push Dataset but got nothing

Hi, i'm new I try to make like https://crawlee.dev/docs/examples/playwright-crawler

but make none data on storage : /

import { Dataset, PlaywrightCrawler, createPlaywrightRouter } from "crawlee";
export const wiki_lexique_de_orgue = createPlaywrightRouter();

wiki_lexique_de_orgue.addDefaultHandler(async ({ enqueueLinks, log }) => {
    log.info(`enqueueing new URLs`);
    await enqueueLinks({
        globs: ['https://fr.wikipedia.org/wiki/Lexique_de_l%27orgue'],
        label: 'detail',
    });
});

wiki_lexique_de_orgue.addHandler('detail', async ({ request, page, log }) => {
    const title = await page.title();
    // const XPATH="xpath=//main/div[3]/div[3]/div[1]/ul/li"
    // const STR=(await page.$$eval() ));
    const data = await page.$$eval('.dxp-node', ($posts: HTMLElement[]) => {
        const scrapedData: { title: string; desc: string}[] = [];
        // We're getting the title, rank and URL of each post on Hacker News.
        $posts.forEach(($post) => {
            scrapedData.push({
                title: $post.querySelector('b')!.innerText,
                desc: $post!.innerText!
            });
        });
        return scrapedData;
    });
    await Dataset.pushData(data);
    log.info(`${title}`, { url: request.loadedUrl });
});

export const WIKI_LEXIQUE_DE_ORGUE = { route: wiki_lexique_de_orgue, start: ["https://fr.wikipedia.org/wiki/Lexique_de_l%27orgue"] }

const crawler = new PlaywrightCrawler({requestHandler: WIKI_LEXIQUE_DE_ORGUE.route,});

await crawler.run(WIKI_LEXIQUE_DE_ORGUE.start);

import { Dataset, PlaywrightCrawler, createPlaywrightRouter } from "crawlee";
export const wiki_lexique_de_orgue = createPlaywrightRouter();

wiki_lexique_de_orgue.addDefaultHandler(async ({ enqueueLinks, log }) => {
    log.info(`enqueueing new URLs`);
    await enqueueLinks({
        globs: ['https://fr.wikipedia.org/wiki/Lexique_de_l%27orgue'],
        label: 'detail',
    });
});

wiki_lexique_de_orgue.addHandler('detail', async ({ request, page, log }) => {
    const title = await page.title();
    // const XPATH="xpath=//main/div[3]/div[3]/div[1]/ul/li"
    // const STR=(await page.$$eval() ));
    const data = await page.$$eval('.dxp-node', ($posts: HTMLElement[]) => {
        const scrapedData: { title: string; desc: string}[] = [];
        // We're getting the title, rank and URL of each post on Hacker News.
        $posts.forEach(($post) => {
            scrapedData.push({
                title: $post.querySelector('b')!.innerText,
                desc: $post!.innerText!
            });
        });
        return scrapedData;
    });
    await Dataset.pushData(data);
    log.info(`${title}`, { url: request.loadedUrl });
});

export const WIKI_LEXIQUE_DE_ORGUE = { route: wiki_lexique_de_orgue, start: ["https://fr.wikipedia.org/wiki/Lexique_de_l%27orgue"] }

const crawler = new PlaywrightCrawler({requestHandler: WIKI_LEXIQUE_DE_ORGUE.route,});

await crawler.run(WIKI_LEXIQUE_DE_ORGUE.start);

I really don't understand how does it work :
- I have url log
- playwright is ok ??????

With ".dxp-node" I expected to fetch 153 text nodes ...

Playwright crawler | Crawlee

This example demonstrates how to use PlaywrightCrawler in combination with RequestQueue to recursively scrape the Hacker News website using headless Chrome / Playwright.

Apify & Crawlee•3y ago•

15 replies

living-lavender

push Dataset but got nothing

Hi, i'm new I try to make like https://crawlee.dev/docs/examples/playwright-crawler

but make none data on storage : /

import { Dataset, PlaywrightCrawler, createPlaywrightRouter } from "crawlee";
export const wiki_lexique_de_orgue = createPlaywrightRouter();

wiki_lexique_de_orgue.addDefaultHandler(async ({ enqueueLinks, log }) => {
    log.info(`enqueueing new URLs`);
    await enqueueLinks({
        globs: ['https://fr.wikipedia.org/wiki/Lexique_de_l%27orgue'],
        label: 'detail',
    });
});

wiki_lexique_de_orgue.addHandler('detail', async ({ request, page, log }) => {
    const title = await page.title();
    // const XPATH="xpath=//main/div[3]/div[3]/div[1]/ul/li"
    // const STR=(await page.$$eval() ));
    const data = await page.$$eval('.dxp-node', ($posts: HTMLElement[]) => {
        const scrapedData: { title: string; desc: string}[] = [];
        // We're getting the title, rank and URL of each post on Hacker News.
        $posts.forEach(($post) => {
            scrapedData.push({
                title: $post.querySelector('b')!.innerText,
                desc: $post!.innerText!
            });
        });
        return scrapedData;
    });
    await Dataset.pushData(data);
    log.info(`${title}`, { url: request.loadedUrl });
});

export const WIKI_LEXIQUE_DE_ORGUE = { route: wiki_lexique_de_orgue, start: ["https://fr.wikipedia.org/wiki/Lexique_de_l%27orgue"] }

const crawler = new PlaywrightCrawler({requestHandler: WIKI_LEXIQUE_DE_ORGUE.route,});

await crawler.run(WIKI_LEXIQUE_DE_ORGUE.start);

import { Dataset, PlaywrightCrawler, createPlaywrightRouter } from "crawlee";
export const wiki_lexique_de_orgue = createPlaywrightRouter();

wiki_lexique_de_orgue.addDefaultHandler(async ({ enqueueLinks, log }) => {
    log.info(`enqueueing new URLs`);
    await enqueueLinks({
        globs: ['https://fr.wikipedia.org/wiki/Lexique_de_l%27orgue'],
        label: 'detail',
    });
});

wiki_lexique_de_orgue.addHandler('detail', async ({ request, page, log }) => {
    const title = await page.title();
    // const XPATH="xpath=//main/div[3]/div[3]/div[1]/ul/li"
    // const STR=(await page.$$eval() ));
    const data = await page.$$eval('.dxp-node', ($posts: HTMLElement[]) => {
        const scrapedData: { title: string; desc: string}[] = [];
        // We're getting the title, rank and URL of each post on Hacker News.
        $posts.forEach(($post) => {
            scrapedData.push({
                title: $post.querySelector('b')!.innerText,
                desc: $post!.innerText!
            });
        });
        return scrapedData;
    });
    await Dataset.pushData(data);
    log.info(`${title}`, { url: request.loadedUrl });
});

export const WIKI_LEXIQUE_DE_ORGUE = { route: wiki_lexique_de_orgue, start: ["https://fr.wikipedia.org/wiki/Lexique_de_l%27orgue"] }

const crawler = new PlaywrightCrawler({requestHandler: WIKI_LEXIQUE_DE_ORGUE.route,});

await crawler.run(WIKI_LEXIQUE_DE_ORGUE.start);

I really don't understand how does it work :
- I have url log
- playwright is ok ??????

With ".dxp-node" I expected to fetch 153 text nodes ...

Playwright crawler | Crawlee

This example demonstrates how to use PlaywrightCrawler in combination with RequestQueue to recursively scrape the Hacker News website using headless Chrome / Playwright.

push Dataset but got nothing

Similar Threads

push Dataset but got nothing

Similar Threads

Similar Threads

Similar Threads