How can I make Playwright browser not skip page if a selector for textContent is not matched

I'm trying to crawl some websites. I know the selector I have should work for some pages, and I want to fall back to

body

body

textContent when the selector doesn't apply. It looks to me like it's actually just failing the whole request here.

Any other options I should pass to the Crawler or to the handler?

const crawler = new PlaywrightCrawler({
    // proxyConfiguration: new ProxyConfiguration({ proxyUrls: ['...'] }),
    requestHandler: router,    
});

const crawler = new PlaywrightCrawler({
    // proxyConfiguration: new ProxyConfiguration({ proxyUrls: ['...'] }),
    requestHandler: router,    
});

router.addHandler('detail', async ({ request, page, log }) => {
    log.info(`processing detail page ${request.loadedUrl}`)
    const title = await page.title();
    const html = await page.content();
    let text;
    try{
        text = await page.textContent(".css-h7put4", {timeout: 5000});
    }
    catch(e){
        console.log("Error getting text content", e);
    }
    if(!text){
        text = await page.textContent("body");
    }

router.addHandler('detail', async ({ request, page, log }) => {
    log.info(`processing detail page ${request.loadedUrl}`)
    const title = await page.title();
    const html = await page.content();
    let text;
    try{
        text = await page.textContent(".css-h7put4", {timeout: 5000});
    }
    catch(e){
        console.log("Error getting text content", e);
    }
    if(!text){
        text = await page.textContent("body");
    }

HonzaS•1/31/23, 4:21 AM

Can you share the error message?

developed-pinkOP•1/31/23, 4:28 AM

yes!

MEE6•1/31/23, 4:28 AM

@NicoNico just advanced to level 1! Thanks for your contributions!

developed-pinkOP•1/31/23, 4:29 AM

developed-pinkOP•1/31/23, 4:29 AM

Separately, I'm noticing it looks like it doesn't crawl all the pages on the domain, I'm curious what I might be doing wrong there too

developed-pinkOP•1/31/23, 4:29 AM

or if it's just erroring out on some and that's what's causing the issue

developed-pinkOP•1/31/23, 4:33 AM

If you're up for hopping on a call Honza, I'd be happy to compensate you for your time to get some of this debugged

HonzaS•1/31/23, 4:53 AM

from the error it looks like it did not find

body

body

as there is default 30000ms timeout but there should be line of the code in the error to be sure
I have never used I am mostly using this https://crawlee.dev/api/playwright-crawler/interface/PlaywrightCrawlingContext#parseWithCheerio
as for crawling all pages I would need to see whole code, generaly if you enqueue the request then it should be processed sooner or later
I am on train now so cannot call

developed-pinkOP•1/31/23, 4:54 AM

oh that's interesting, just grab html and parse with cheerio

developed-pinkOP•1/31/23, 4:54 AM

ok let me try that

developed-pinkOP•1/31/23, 4:55 AM

yah I see it error out on like 3 pages, and then it stops

developed-pinkOP•1/31/23, 4:55 AM

wondering if there is some other settings like maxrequests, or something to that effect

developed-pinkOP•1/31/23, 4:55 AM

but I haven't found anything in the docs that seemed relevant

developed-pinkOP•1/31/23, 4:56 AM

just made the cheerio change and re-running now

developed-pinkOP•1/31/23, 4:56 AM

looking good

developed-pinkOP•1/31/23, 4:56 AM

thank you!

developed-pinkOP•1/31/23, 4:57 AM

hmm it definitely still stopped

developed-pinkOP•1/31/23, 4:57 AM

i.e. didn't get all links

developed-pinkOP•1/31/23, 4:57 AM

full code is

HonzaS•1/31/23, 4:58 AM

there is this https://crawlee.dev/docs/next/introduction/adding-urls#limit-your-crawls-with-maxrequestspercrawl
but if it is not set then it should not limit anything

developed-pinkOP•1/31/23, 5:01 AM

router.addDefaultHandler(async ({ enqueueLinks, log }) => {
log.info();
await enqueueLinks({
strategy: EnqueueStrategy.SameDomain,
label: 'detail',
});
});

developed-pinkOP•1/31/23, 5:05 AM

seems to stop at 30 pages, but there's definitely more than that at: https://docs.snyk.io/

Snyk User Documentation

HonzaS•1/31/23, 5:16 AM

so it looks like it enqueues just those 30 links
you can log what links it enqueued
or you can try to enqueue it by hand, just get links you want and enqueue them, that way you will have more control

developed-pinkOP•1/31/23, 5:18 AM

interesting, any idea why it only enqueues those 30?

MEE6•1/31/23, 5:18 AM

@NicoNico just advanced to level 2! Thanks for your contributions!

developed-pinkOP•1/31/23, 5:19 AM

my sense from the docs was that it should include all the links on the page if I do something like:

HonzaS•1/31/23, 5:19 AM

there is 35 links on the page as I see

developed-pinkOP•1/31/23, 5:19 AM

but does it not do it recursively?

developed-pinkOP•1/31/23, 5:19 AM

i.e. does it not crawl beyond the first level of depth?

developed-pinkOP•1/31/23, 5:19 AM

I assume it would continue spidering on to the next page

developed-pinkOP•1/31/23, 5:20 AM

ah I see

HonzaS•1/31/23, 5:20 AM

nope as far as I know you would need to implement that logic

developed-pinkOP•1/31/23, 5:20 AM

the handler is taking precedent and then I need to enque there too

HonzaS•1/31/23, 5:20 AM

yeah, you can put there the same code

developed-pinkOP•1/31/23, 5:20 AM

trying now

developed-pinkOP•1/31/23, 5:21 AM

thanks again for your help HonzaS

developed-pinkOP•1/31/23, 5:22 AM

I'd love to send you a token of my appreciation for your help if you send me a DM or you have a link

developed-pinkOP•1/31/23, 5:22 AM

that worked by the way, problem was I wasn't enqueing on the details handler

developed-pinkOP•1/31/23, 5:23 AM

thanks again!

HonzaS•1/31/23, 5:24 AM

yeah, if you want to go through all the pages you need to enqueue on all pages to find all the links
no problem I have nothing to do on the train anyway

developed-pinkOP•1/31/23, 5:30 AM

it's late where I am (west coast USA), but I'd love some help tomorrow trying to deploy this as a service to the platform - which I haven't done before, let me know if you think you'd have some time (I'd compensate you)

developed-pinkOP•1/31/23, 5:31 AM

I'm still running it locally

developed-pinkOP•1/31/23, 5:31 AM

your suggestion def fixed my issues!

HonzaS•1/31/23, 5:50 AM

I am using this https://www.npmjs.com/package/apify-cli
I am developing locally mostly too and then just write command . The program is then pushed to the platform and built there.
Let me know if you run into any issues.