Pupeteer unable to find element (dev tools show the element)

Hello everyone,

I'm currently working on a web scraping tool that extracts data from the following web page: https://www.handelsregister.de/rp_web/documents-dk.xhtml.
However, I'm running into some unexpected issues and I'm not sure how to debug them.
Here's a brief overview of the issue:
- I'm using Puppeteer to scrape data from the web page.
- The tool is able to navigate to the correct page and locate the element that I want to click.
- However, when I try to extract the element using document.querySelectorAll(), I'm not getting the expected results.
- Specifically, I'm using the following code to extract the first toggle span:

const firstToggleSpan = await page.evaluate(() => {
  const toggleSpans = document.querySelectorAll(
    "span.ui-tree-toggler.ui-icon.ui-icon-triangle-1-s",
  );
  return toggleSpans[0]; // prints `{}` to   the server logs
});

const firstToggleSpan = await page.evaluate(() => {
  const toggleSpans = document.querySelectorAll(
    "span.ui-tree-toggler.ui-icon.ui-icon-triangle-1-s",
  );
  return toggleSpans[0]; // prints `{}` to   the server logs
});

- However, firstToggleSpan is an empty object,
- whereas running the same code in the Puppeteer DevTools console, it returns the element that I want to click.

How can I debug pupeteer's page selectors and why doesn't it behave as the querySelector in the dev tools?

Apify & Crawlee•3y ago•

2 replies

awake-maroon

Pupeteer unable to find element (dev tools show the element)

const firstToggleSpan = await page.evaluate(() => {
  const toggleSpans = document.querySelectorAll(
    "span.ui-tree-toggler.ui-icon.ui-icon-triangle-1-s",
  );
  return toggleSpans[0]; // prints `{}` to   the server logs
});

const firstToggleSpan = await page.evaluate(() => {
  const toggleSpans = document.querySelectorAll(
    "span.ui-tree-toggler.ui-icon.ui-icon-triangle-1-s",
  );
  return toggleSpans[0]; // prints `{}` to   the server logs
});

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,851 Members

Join

Pupeteer unable to find element (dev tools show the element)

Pupeteer unable to find element (dev tools show the element)

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads