I want to fetch some JSON with an
I want to fetch some JSON with an HttpCrawler and then from the JSON fetch a url with CheerioCrawler. How to combine these two?
3 Replies
What is the point? Just use only the cheerio
Let's say that for each result, I need to scrape 3 pages. Each time base, /about & /contact. How would you go about that? Nest crawlers inside each other or enqueue the links and keep track of intermediary results in KV storage or something else?
one crawler, keep track of already scraped in userData of the request
meaning first enqueue base, then in base handled enqueue about and put the results from the base in the userData and so on and in conctact put the whole result to the dataset