How to Avoid Duplicate Leads When Using Apify for Long-Term Lead Generation?
General💸Pricing🛠️APIHey, I have a question about using Apify for long-term lead generation.
I'm planning to use Apify to fetch around 1,000 leads per day based on specific locations and job positions. I expect to run this process continuously for about 4–5 months to build and enrich my database.
My main concern is duplicate profiles across different runs.
For example, let's say I've already collected 10,000 profiles. The next day, I request another 1,000 profiles matching the same criteria.
How does Apify ensure that those 1,000 results haven't already been returned in previous runs?
If 900 of those profiles are duplicates and only 100 are new, I would effectively be paying for 1,000 results while only adding 100 new leads to my database.
Is there a built-in mechanism to prevent previously retrieved profiles from being fetched again across multiple runs?
Ideally, I would like to maintain a list of already retrieved profile IDs/URLs and have Apify exclude them before scraping or charging for the results.
Does Apify support something like this, or would the deduplication need to be handled on my side?
I'm planning to use Apify to fetch around 1,000 leads per day based on specific locations and job positions. I expect to run this process continuously for about 4–5 months to build and enrich my database.
My main concern is duplicate profiles across different runs.
For example, let's say I've already collected 10,000 profiles. The next day, I request another 1,000 profiles matching the same criteria.
How does Apify ensure that those 1,000 results haven't already been returned in previous runs?
If 900 of those profiles are duplicates and only 100 are new, I would effectively be paying for 1,000 results while only adding 100 new leads to my database.
Is there a built-in mechanism to prevent previously retrieved profiles from being fetched again across multiple runs?
Ideally, I would like to maintain a list of already retrieved profile IDs/URLs and have Apify exclude them before scraping or charging for the results.
Does Apify support something like this, or would the deduplication need to be handled on my side?
