Excluding urls from enqueue links
I've scraped a website which includes all links found using globs pattern
eg.
Now I want to scrape same website but I don't want to scrape links that I've already visited.
My flow is that I provide root url and I have a root route handler which enqueues links using pattern.
How could I do same thing again, but I want to exclude all the links I've scraped in previous run?
ps. This website adds new content weekly.
eg.
Now I want to scrape same website but I don't want to scrape links that I've already visited.
My flow is that I provide root url and I have a root route handler which enqueues links using pattern.
How could I do same thing again, but I want to exclude all the links I've scraped in previous run?
ps. This website adds new content weekly.
