living-lavenderL

how deep can website-content-crawler go?

hello guys I have a website that is composed of 3 million pages, the website homepage is like google so the crawler has to enter all the search results one by one and scrape the content inside each of them, can website-content-crawler do all that automatically or do I have to give it the links to those 3 million pages?
also can I customize what to scrape inside each of those links? like give it div id of the container?
Was this page helpful?