Elements not rendering
Hi, I am trying to crawl a page (it's a very small crawl, I used to do it manually every week).
I am using the PlaywrightCrawler.
I am running into a problem, when visiting the site I want to crawl, the elements that are most important to crawl are not appearing.
They are not hidden with CSS or something, they are just literally not in the DOM. The website uses server side rendering.
I thought that it may have to do something with screen size. So I decided to disable headless, increase browser and page viewport to 1920x1080 and try it again. However, the elements were still not being shown.
I tried adjusting my viewport in a different browser (not crawling, just browsing), and it shows the data on every viewport (even the tiniest I can make it).
Next I tried visiting the site on my phone, and the data was not showing. Could it be that it detects the crawler as a phone or something? I didn't turn anything regarding this on.
I am using the PlaywrightCrawler.
I am running into a problem, when visiting the site I want to crawl, the elements that are most important to crawl are not appearing.
They are not hidden with CSS or something, they are just literally not in the DOM. The website uses server side rendering.
I thought that it may have to do something with screen size. So I decided to disable headless, increase browser and page viewport to 1920x1080 and try it again. However, the elements were still not being shown.
I tried adjusting my viewport in a different browser (not crawling, just browsing), and it shows the data on every viewport (even the tiniest I can make it).
Next I tried visiting the site on my phone, and the data was not showing. Could it be that it detects the crawler as a phone or something? I didn't turn anything regarding this on.
