Ffoolish-indigo
structure for broad shallow scrape
Hi, I have a question about the best way to structure my solution to this problem to best take advantage of apify infrastructure and services. I need to scrape once per day a dataset off of 500 distinct domains, but each only 1-2 pages. The selectors for the items are different for almost all sites. The two extremes are 500 separate actors and one actor with a hashmap of domains to selectors for that domain. I want to be able to track when a domain has broken. What’s the best way to structure this that aligns myself with apifys infra?
This is the official developer community of Apify and Crawlee.
14,091Members
Resources
Recent Announcements
Similar Threads
Was this page helpful?
