Help with website-content-crawler for specific website
Hi all, I'm an ML engineer but complete beginner using Apify. It looks great in theory but for my first project I ran into the roadblock that I cannot crawl this website properly: https://georgian.io/resources/
There are a bunch of articles and blogs there for which I want the text but running the web-crawler only returns the main page without any sub-links, regardless of the max-depth I specify.
I essentially followed the langchain tutorial here: https://apify.com/apify/website-content-crawler.
Simply replacing the url. I then also manually set the max depth in the website for the same call and still got a single page as output.
Any advise? Thanks!!
There are a bunch of articles and blogs there for which I want the text but running the web-crawler only returns the main page without any sub-links, regardless of the max-depth I specify.
I essentially followed the langchain tutorial here: https://apify.com/apify/website-content-crawler.
Simply replacing the url. I then also manually set the max depth in the website for the same call and still got a single page as output.
Any advise? Thanks!!
Georgian
Find resources from Georgian by topic (applied AI, trust, blockchain, conversational AI and more) or by type of content (video, podcast, white papers...)
