Concurrent Runs
On the pricing page https://apify.com/pricing it has a row for concurrent runs but I cant find any documentation on how to do this.
I am trying to scrape Zillow to create a table/matrix of 9x14=126 cells where each cell is the result (a number) from he API response with parameters on Geography, Time, Status, Size. Each API call currently takes 5 seconds so doing them synchronously takes 126 api calls x 5 sec=630 seconds or 10 MINUTES to complete!
If I could send 32 concurrently on my $49 paid plan it should be way faster 126/32=3.94 x 5 seconds=20 seconds to complete vs 630 seconds/10min
I guess one way is to somehow be able to send 32 separate requests simultaneously and do this 4x as described above.
Another approach would be to somehow send the API request to Apify in a single POST but have the search terms for each 126 cells as an array but not sure if this is possible or how to do it. Also not clear this would be faster
Here is a sample of what I want my final output to look like just trying to speed this up:
I am trying to scrape Zillow to create a table/matrix of 9x14=126 cells where each cell is the result (a number) from he API response with parameters on Geography, Time, Status, Size. Each API call currently takes 5 seconds so doing them synchronously takes 126 api calls x 5 sec=630 seconds or 10 MINUTES to complete!
If I could send 32 concurrently on my $49 paid plan it should be way faster 126/32=3.94 x 5 seconds=20 seconds to complete vs 630 seconds/10min
I guess one way is to somehow be able to send 32 separate requests simultaneously and do this 4x as described above.
Another approach would be to somehow send the API request to Apify in a single POST but have the search terms for each 126 cells as an array but not sure if this is possible or how to do it. Also not clear this would be faster
Here is a sample of what I want my final output to look like just trying to speed this up:
Apify
Extract value from the web with Apify. Flexible platform — flexible pricing. Free plan available, no credit card required.
