How can I make Playwright browser not skip page if a selector for textContent is not matched

I'm trying to crawl some websites. I know the selector I have should work for some pages, and I want to fall back to

body

body

textContent when the selector doesn't apply. It looks to me like it's actually just failing the whole request here.

Any other options I should pass to the Crawler or to the handler?

const crawler = new PlaywrightCrawler({
    // proxyConfiguration: new ProxyConfiguration({ proxyUrls: ['...'] }),
    requestHandler: router,    
});

const crawler = new PlaywrightCrawler({
    // proxyConfiguration: new ProxyConfiguration({ proxyUrls: ['...'] }),
    requestHandler: router,    
});

router.addHandler('detail', async ({ request, page, log }) => {
    log.info(`processing detail page ${request.loadedUrl}`)
    const title = await page.title();
    const html = await page.content();
    let text;
    try{
        text = await page.textContent(".css-h7put4", {timeout: 5000});
    }
    catch(e){
        console.log("Error getting text content", e);
    }
    if(!text){
        text = await page.textContent("body");
    }

router.addHandler('detail', async ({ request, page, log }) => {
    log.info(`processing detail page ${request.loadedUrl}`)
    const title = await page.title();
    const html = await page.content();
    let text;
    try{
        text = await page.textContent(".css-h7put4", {timeout: 5000});
    }
    catch(e){
        console.log("Error getting text content", e);
    }
    if(!text){
        text = await page.textContent("body");
    }

Apify & Crawlee•4y ago•

44 replies

developed-pink

How can I make Playwright browser not skip page if a selector for textContent is not matched

I'm trying to crawl some websites. I know the selector I have should work for some pages, and I want to fall back to

body

body

textContent when the selector doesn't apply. It looks to me like it's actually just failing the whole request here.

Any other options I should pass to the Crawler or to the handler?

const crawler = new PlaywrightCrawler({
    // proxyConfiguration: new ProxyConfiguration({ proxyUrls: ['...'] }),
    requestHandler: router,    
});

const crawler = new PlaywrightCrawler({
    // proxyConfiguration: new ProxyConfiguration({ proxyUrls: ['...'] }),
    requestHandler: router,    
});

router.addHandler('detail', async ({ request, page, log }) => {
    log.info(`processing detail page ${request.loadedUrl}`)
    const title = await page.title();
    const html = await page.content();
    let text;
    try{
        text = await page.textContent(".css-h7put4", {timeout: 5000});
    }
    catch(e){
        console.log("Error getting text content", e);
    }
    if(!text){
        text = await page.textContent("body");
    }

router.addHandler('detail', async ({ request, page, log }) => {
    log.info(`processing detail page ${request.loadedUrl}`)
    const title = await page.title();
    const html = await page.content();
    let text;
    try{
        text = await page.textContent(".css-h7put4", {timeout: 5000});
    }
    catch(e){
        console.log("Error getting text content", e);
    }
    if(!text){
        text = await page.textContent("body");
    }

How can I make Playwright browser not skip page if a selector for textContent is not matched

Similar Threads

How can I make Playwright browser not skip page if a selector for textContent is not matched

Similar Threads

Similar Threads

Similar Threads