Keeping track of the parent page with PlaywrightCrawler

Hi! I'm using Crawlee as an e2e test for broken links and generated diagrams in our documentation website. So far it's been successful and the only thing I'm missing is figuring out what page actually contained the broken link.

For example, this is the snippet I use to find pages that display the 404 message:

async requestHandler({ request, page, enqueueLinks, log }) {
    // check if Docusaurus handled 404
    const isDocusaurus404 = await page
      .locator(".terminal-body")
      .getByText("404")
      .count();

    if (isDocusaurus404) {
      console.log({ url: page.url()});      
    }

    await enqueueLinks();
  },

async requestHandler({ request, page, enqueueLinks, log }) {
    // check if Docusaurus handled 404
    const isDocusaurus404 = await page
      .locator(".terminal-body")
      .getByText("404")
      .count();

    if (isDocusaurus404) {
      console.log({ url: page.url()});      
    }

    await enqueueLinks();
  },

This will log the actual URL that does not exist, but I can't tell which page contained that URL. What's the easiest way to find this information? Sort of like a History API? Thanks!

Apify & Crawlee•4y ago•

6 replies

embarrassing-maroon

Keeping track of the parent page with PlaywrightCrawler

async requestHandler({ request, page, enqueueLinks, log }) {
    // check if Docusaurus handled 404
    const isDocusaurus404 = await page
      .locator(".terminal-body")
      .getByText("404")
      .count();

    if (isDocusaurus404) {
      console.log({ url: page.url()});      
    }

    await enqueueLinks();
  },

async requestHandler({ request, page, enqueueLinks, log }) {
    // check if Docusaurus handled 404
    const isDocusaurus404 = await page
      .locator(".terminal-body")
      .getByText("404")
      .count();

    if (isDocusaurus404) {
      console.log({ url: page.url()});      
    }

    await enqueueLinks();
  },

This will log the actual URL that does not exist, but I can't tell which page contained that URL. What's the easiest way to find this information? Sort of like a History API? Thanks!

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,851 Members

Join

Keeping track of the parent page with PlaywrightCrawler

Keeping track of the parent page with PlaywrightCrawler

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads