Ffull-green
Adding multiple requestHandler as a workflow
I am making a crawler which scrapes socials from a website. So the current workflow of the crawler I want is:
Given a list of urls and names of the business websites
- It first goes to the homepage and looks for socials there
- then enques the links which includes path like contact or about and crawl that and then moves to the other url
The thing I want to add is after it finishes with scraping homepage, contact or about (if any). I have another class which takes page as a params and has a function
Another thing I want to do is make the datasets returns the results instead saving it individually. The problem with that is it saves duplicates. For example if the socials are in the footer, and it also has about page. then socials are saved in duplicates twice which I want to remove that
Given a list of urls and names of the business websites
- It first goes to the homepage and looks for socials there
- then enques the links which includes path like contact or about and crawl that and then moves to the other url
The thing I want to add is after it finishes with scraping homepage, contact or about (if any). I have another class which takes page as a params and has a function
getGoogleResultsgetGoogleResults (get top search results when enter a business name). I want to invoke that requestHandler which invokes getGoogleResults(query)getGoogleResults(query) function and add the results to the Datasets.Another thing I want to do is make the datasets returns the results instead saving it individually. The problem with that is it saves duplicates. For example if the socials are in the footer, and it also has about page. then socials are saved in duplicates twice which I want to remove that
