Ffaint-white
Question about userData as option in enqueueLinks
Scraping a few pages of a forum. Some infos are in the threadlist page and other are into the thread. In the end these information has to be loaded in a pg db.
Example of thread list, information scraped href to feed in the enqueue links, views and replies
While in the post I generally need title, post body, author, date etc...
Needing some data that's outside and inside, I'd find useful the userData option in enqueueLinks(). Right now my scraper runs on cheerios and the main focus is on this piece of code. Since replies and views are different for every thread, how could I pass these information since links gets passed in bulk with a string[]
I see multiple options, though nothing looks so neat:
- make x multiple enqueueLinks call for x thread. Passing everytime an array of 1 element with the related userdata
- somehow storing locally, using dataset or someother key-value as temporary storage
The desirable option would be having the chance of sending two parallel vectors. Getting the index of the current link being runned and recover the related data from those arrays. Though I don't understand if there's a way to get the index.
I would like to hear the opinion of someone more expert of me on this thing. Thankies ^^
Example of thread list, information scraped href to feed in the enqueue links, views and replies
While in the post I generally need title, post body, author, date etc...
Needing some data that's outside and inside, I'd find useful the userData option in enqueueLinks(). Right now my scraper runs on cheerios and the main focus is on this piece of code. Since replies and views are different for every thread, how could I pass these information since links gets passed in bulk with a string[]
I see multiple options, though nothing looks so neat:
- make x multiple enqueueLinks call for x thread. Passing everytime an array of 1 element with the related userdata
- somehow storing locally, using dataset or someother key-value as temporary storage
The desirable option would be having the chance of sending two parallel vectors. Getting the index of the current link being runned and recover the related data from those arrays. Though I don't understand if there's a way to get the index.
I would like to hear the opinion of someone more expert of me on this thing. Thankies ^^
This is the official developer community of Apify and Crawlee.
14,091Members
Resources
Recent Announcements
Similar Threads
Was this page helpful?
