Eendless-jade
Need to have an identifier to link back scrapped data to main database
Hi, I am currently using Apify scrappers to scrape IG account information and FB page ad library information.
The goal of using the scrapper is to get information about my leads and import back that information to my main leads database. However, depending on the scrapper, I sometime don't have an "identifier" to link back the data from the scraper, back to the correct lead in my main data base.
The way I link the information I scrapped back to the right lead is using "lookup". for example: I look up the domain url in both my main database and the scrapped data to link the right information to the sheet.
However, to scrap FB page ad library, I am using the URL of the FB page. Most of the time, the url is formatted correctly, but sometimes it is not (it may have "fr." or other /... ). And when returning the data from apify, it always gives me back the correctly formatted data (the FB page URL) Therefore, I cannot "lookup" all of those leads that had a FB page url that wasn't formatted correctly in my main database.
Question #1. Is there a way to have a ID or the "input data" in the ouput from apify so that I can link the scrapped data back to the correct lead in the main database.
Question #2. Is there a better way to link those datasets together?
The goal of using the scrapper is to get information about my leads and import back that information to my main leads database. However, depending on the scrapper, I sometime don't have an "identifier" to link back the data from the scraper, back to the correct lead in my main data base.
The way I link the information I scrapped back to the right lead is using "lookup". for example: I look up the domain url in both my main database and the scrapped data to link the right information to the sheet.
However, to scrap FB page ad library, I am using the URL of the FB page. Most of the time, the url is formatted correctly, but sometimes it is not (it may have "fr." or other /... ). And when returning the data from apify, it always gives me back the correctly formatted data (the FB page URL) Therefore, I cannot "lookup" all of those leads that had a FB page url that wasn't formatted correctly in my main database.
Question #1. Is there a way to have a ID or the "input data" in the ouput from apify so that I can link the scrapped data back to the correct lead in the main database.
Question #2. Is there a better way to link those datasets together?
This is the official developer community of Apify and Crawlee.
14,091Members
Resources
Recent Announcements
Similar Threads
Was this page helpful?
