specific-silverS
Apify & Crawlee2y ago
2 replies
specific-silver

Hello everyone, Is there a way to avoid

Hello everyone, Is there a way to avoid scraping same pages even if the crawler is restared ??? because I'm currently working on a news website crawler, However, with each run of the scraper, I'm encountering up to 80% duplicated news from previous runs. Any suggestions on how to address this issue effectively?
Was this page helpful?