Ignore previously crawled URLs
Is there a simple way to go about ignoring previously crawled URLs? Or should I implement logic for detecting if a URL has been previously crawled and skip it? My current approach is to store the items in a separate database, and then use https://crawlee.dev/docs/introduction/adding-urls#transform-requests transform requests to determine whether or not to crawl the link
