Not generating jsons but crawling
I have scraped so far 300k links and generated json files. My crawler still crawls through the links as per my labels and globs patterns but it isn’t generating json files.
I have 2 questions here:
1. Does the crawler.rub() take not only the url but also a label like {url: xyz.com, label:”website”}
2. If I can fix this and start scraping again I don’t want the 300k to get started again, instead I would want only the delta.
Any help would be much much appreciated!
Note: I built my crawler in Typescript
I have 2 questions here:
1. Does the crawler.rub() take not only the url but also a label like {url: xyz.com, label:”website”}
2. If I can fix this and start scraping again I don’t want the 300k to get started again, instead I would want only the delta.
Any help would be much much appreciated!
Note: I built my crawler in Typescript
