I use this to check the currently used

I use this to check the currently used dataset:

await (await Actor.openDataset()).getInfo()

await (await Actor.openDataset()).getInfo()

. However, often times it is not up to date (I feel like some eventually consistent problem is playing). Is it possible of fetching the data when it is consistent? Currently I'm just waiting 3 seconds before fetching, but that feels "hacky".

azzouzana•4/21/25, 11:08 AM

That's a known behavior.

They mention that it takes up to 5 seconds interval to update the data.

You might want to get the dataset items and check their length to have realtime count

azzouzana•4/21/25, 11:08 AM

azzouzana•4/21/25, 11:11 AM

Or you can track it manually using some variables which you'd ++ if adding to the dataset is OK

Louis DeconinckOP•4/21/25, 1:19 PM

Thanks for the info @azzouzana! Link to docs: https://docs.apify.com/api/v2/dataset-get

I've switched to which works. Is this a better approach? Does it have downsides? If there are no downsides, why would you ever use ?

Link docs: https://docs.apify.com/sdk/js/reference/interface/DatasetContent

The documentation mentions the existence of both

count

count

and . But I'm not sure what the actual difference is?

count: Count of dataset entries returned in this set.
total: Total count of entries in the dataset.

Get dataset | Apify Documentation

ClientsReturns dataset object for given dataset ID.

DatasetContent | API | SDK for JavaScript | Apify Documentation

azzouzana•4/21/25, 1:37 PM

@Louis Deconinck count is the number of entries in that part of the data you fetched. The downside is that there's some pagination involved & it's relatively slower and could become expensive if done frequently, I remember somewhere that the max you can get in a is 10000 & now it seems it has been updated to 250000 https://crawlee.dev/js/api/core/interface/DatasetDataOptions#limit so please experiment with it if needed) vs total which should return the total number of entries in the dataset. I believe getInfo().itemCount is to be used when you don't have to worry about caching/realtime issues as it's some pre-calculated metadata so it's cheap.

For example, if you have a dataset with 857 items
The response of

is
items

I remember for the best possible performance is to do something like so you don't fetch all the dataset entries

Louis DeconinckOP•4/21/25, 1:47 PM

Thanks @azzouzana, worked perfectly!