Advice concerning run limits and cost optimization
Hi everyone
I am currently designing an application that will scrap data on social networks and I am currently considering Apify to do this.
However, I have some technical questions, mainly concerning the limits of the actors' runs and cost optimization. After reading the documentation thoroughly, some of my questions remain unanswered.
To give you a bit of context, the app will scrap facebook and instagram pages and posts periodically. The number of pages to scrap will be of the 1000 magnitude, and should have a linear growth with time.
What I am trying to find is a good compromise of batch size for running the page scrapers
Here is a list of the questions I still have after reading the documentation:
-> In the pricing documentation (https://apify.com/pricing), it's mentioned that there is a concurrent run limit. However, I can't find any information about what happens when this limit is reached. If a request more runs than the limit, are they queued ? Or will the api respond with an error message? Moreover, the documentation is not consistent about the value of this limit (it is different here: https://docs.apify.com/platform/limits)
-> In the price optimization section (https://help.apify.com/en/articles/3470975-how-to-estimate-compute-unit-usage-for-your-project), it is said that it's more price efficient to run few big runs rather than many small runs. However, I can't find information concerning what is 'Too big'. Is there any limits concerning how big a run can be (like the number of urls passed in input, the size of the run result or something else ?).
Thank you for your help,
Maxim
I am currently designing an application that will scrap data on social networks and I am currently considering Apify to do this.
However, I have some technical questions, mainly concerning the limits of the actors' runs and cost optimization. After reading the documentation thoroughly, some of my questions remain unanswered.
To give you a bit of context, the app will scrap facebook and instagram pages and posts periodically. The number of pages to scrap will be of the 1000 magnitude, and should have a linear growth with time.
What I am trying to find is a good compromise of batch size for running the page scrapers
Here is a list of the questions I still have after reading the documentation:
-> In the pricing documentation (https://apify.com/pricing), it's mentioned that there is a concurrent run limit. However, I can't find any information about what happens when this limit is reached. If a request more runs than the limit, are they queued ? Or will the api respond with an error message? Moreover, the documentation is not consistent about the value of this limit (it is different here: https://docs.apify.com/platform/limits)
-> In the price optimization section (https://help.apify.com/en/articles/3470975-how-to-estimate-compute-unit-usage-for-your-project), it is said that it's more price efficient to run few big runs rather than many small runs. However, I can't find information concerning what is 'Too big'. Is there any limits concerning how big a run can be (like the number of urls passed in input, the size of the run result or something else ?).
Thank you for your help,
Maxim
Need to estimate the long-term costs of a project? Read this article to understand what affects performance.
