Pupeteer unable to find element (dev tools show the element)

Hello everyone,

I'm currently working on a web scraping tool that extracts data from the following web page: https://www.handelsregister.de/rp_web/documents-dk.xhtml.
However, I'm running into some unexpected issues and I'm not sure how to debug them.
Here's a brief overview of the issue:
- I'm using Puppeteer to scrape data from the web page.
- The tool is able to navigate to the correct page and locate the element that I want to click.
- However, when I try to extract the element using document.querySelectorAll(), I'm not getting the expected results.
- Specifically, I'm using the following code to extract the first toggle span:

const firstToggleSpan = await page.evaluate(() => {
  const toggleSpans = document.querySelectorAll(
    "span.ui-tree-toggler.ui-icon.ui-icon-triangle-1-s",
  );
  return toggleSpans[0]; // prints `{}` to   the server logs
});

const firstToggleSpan = await page.evaluate(() => {
  const toggleSpans = document.querySelectorAll(
    "span.ui-tree-toggler.ui-icon.ui-icon-triangle-1-s",
  );
  return toggleSpans[0]; // prints `{}` to   the server logs
});

- However, firstToggleSpan is an empty object,
- whereas running the same code in the Puppeteer DevTools console, it returns the element that I want to click.

How can I debug pupeteer's page selectors and why doesn't it behave as the querySelector in the dev tools?

Pupeteer unable to find element (dev tools show the element)

const firstToggleSpan = await page.evaluate(() => {
  const toggleSpans = document.querySelectorAll(
    "span.ui-tree-toggler.ui-icon.ui-icon-triangle-1-s",
  );
  return toggleSpans[0]; // prints `{}` to   the server logs
});

const firstToggleSpan = await page.evaluate(() => {
  const toggleSpans = document.querySelectorAll(
    "span.ui-tree-toggler.ui-icon.ui-icon-triangle-1-s",
  );
  return toggleSpans[0]; // prints `{}` to   the server logs
});

Pupeteer unable to find element (dev tools show the element)

Pupeteer unable to find element (dev tools show the element)

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads