exotic-emerald•2y ago
Is it better to separate scraper by domains?
I have about 10 different domains that I want to crawl. Each one of those domain has:
- List view
- Detail view
Is it better to separate those scrapers? I don't need high concurrency. This is a low workload. But how does the route handler look like?
2 Replies
vicious-gold•2y ago
You can do this by creating a router with labels and passing a label to each request, here is one resource you can use: :
https://docs.apify.com/academy/web-scraping-for-beginners/challenge/scraping-amazon
Scraping Amazon | Academy | Apify Documentation
Before you build your first web scraper with Crawlee, it is important to understand the concept of modularity in programming.
exotic-emeraldOP•2y ago
Ohh okay, so basically just namespacing it with labels. Got it.