scraping at scale
How should I structure my crawler when scraping possibly 100s of different sites with different structures, handling multiple requests at once in Crawlee
SWell, I am implementing something similar... 30-40 sites but with SIMILAR structure (if the structure of your sites is different -> you are implementing something like google/bing - king of generic web crawler)
listOfTopics: h1 > div.list > div
...
xyz987.com:
listOfTopics: div.bigListClass > div > p
....
- You might use something like an external message queue, we discussed it here and in few other places:
crawlee-jsExternal request queue + external result storage, Crawlee as daemon process - how to implement it?
- you can create one big config file (YML, JSON...) describing "where-to-find-what on each site"
Example:
listOfTopics: h1 > div.list > div
...
xyz987.com:
listOfTopics: div.bigListClass > div > p
....
Ethank you
M@harish just advanced to level 1! Thanks for your contributions! 