extended-salmon•4w ago

For all Stores i dont know yet there is

For all Stores i dont know yet there is more than 600k stores but for the first 10k stores my json data is 7gb

4 Replies

aciku•4w ago

API data push to a dataset is capped at 200 requests per second to avoid overloading our servers.

API data push to a dataset is capped at 200 requests per second to avoid overloading our servers.

Data storage formats that use tabulation (like HTML, CSV, and EXCEL) are limited to a maximum of 3000 columns. Data exceeding this limit will not be retrieved.

The pushData()method is constrained by the receiving API's size limit. It accepts objects with JSON size under 9MB. While individual objects within an array must not exceed 9MB, the overall size has no restriction.

The maximum length for dataset names is 63 characters.

Rate limiting
The rate limit for pushing data to a dataset through the API is capped at 200 requests per second for each dataset, a measure to prevent overloading Apify servers.

For all other dataset API endpoints , the rate limit is 30 requests per second for each dataset.

Check out the API documentation for more information and guidance on actions to take if you exceed these rate limits.

Data storage formats that use tabulation (like HTML, CSV, and EXCEL) are limited to a maximum of 3000 columns. Data exceeding this limit will not be retrieved.

The pushData()method is constrained by the receiving API's size limit. It accepts objects with JSON size under 9MB. While individual objects within an array must not exceed 9MB, the overall size has no restriction.

The maximum length for dataset names is 63 characters.

Rate limiting
The rate limit for pushing data to a dataset through the API is capped at 200 requests per second for each dataset, a measure to prevent overloading Apify servers.

For all other dataset API endpoints , the rate limit is 30 requests per second for each dataset.

Check out the API documentation for more information and guidance on actions to take if you exceed these rate limits.

https://docs.apify.com/platform/storage/dataset The documentation isn’t entirely clear and could use more detail. I think there’s an inconsistency in the documentation. In one place it states a different limit, but in the API section it says the maximum dataset request payload is 5 MB. Am I missing something? https://docs.apify.com/api/v2/dataset-items-post

aciku•4w ago

So it seems the limit is 9 MB per request.

extended-salmon•4w ago

interesting. I recall it being 10MB some time back....

Lukas Krivka•4w ago

It is 9.4MB. You should push it to Apify Dataset, just push it as array of products, not one giant object. Datasets have infinite size, we had datasets with hundreds of millions of products. If you store it raw to S3 or other clouds, it will be a bit cheaper but you will loose the dataset features. You could also have 1 dataset per store or group them somehow.

For all Stores i dont know yet there is

Did you find this page helpful?