Pprimary-violet
Efficiency / Cost management of multiple smaller actors vs. one large actor
I'm using the clockworks tiktok scraper and had dilemna on whether to use 8 actors with 4gb RAM each, spreading out pages to scrape evenly accross the actors vs. 1 large 32gb actor for all the pages.
It seems like from trying out the scrape its far cheaper (not much faster though), to spread out the jobs accross 8 actors (we're looking at $10-12 vs. $3-4). I was adviced from the team to try one large actor since it should scale okay, but seems like my findings are different?
Am I doing something wrong here, or is it just that this scrape is better with me batching the jobs.
It seems like from trying out the scrape its far cheaper (not much faster though), to spread out the jobs accross 8 actors (we're looking at $10-12 vs. $3-4). I was adviced from the team to try one large actor since it should scale okay, but seems like my findings are different?
Am I doing something wrong here, or is it just that this scrape is better with me batching the jobs.
