Bbrilliant-lime
Saving scraped data from dynamic URLs using Crawlee in an Express Server?
Hello all.
I've been trying to build an app that triggers a scraping job when the api is hit.
The initial endpoint hits a crawlee router which has 2 handlers. one for the url-list scraping and the other for scraping the detail from each of the detail-page. (the url-list handler enqueues the next url-list page to url-list handler too btw)
I'm saving the data from each of these scrapes inside a KVstore but I want a way to save all the data in the KV store related to a particular job into database.
The attached screenshots are the MRE snippets from my code.
I've been trying to build an app that triggers a scraping job when the api is hit.
The initial endpoint hits a crawlee router which has 2 handlers. one for the url-list scraping and the other for scraping the detail from each of the detail-page. (the url-list handler enqueues the next url-list page to url-list handler too btw)
I'm saving the data from each of these scrapes inside a KVstore but I want a way to save all the data in the KV store related to a particular job into database.
The attached screenshots are the MRE snippets from my code.
This is the official developer community of Apify and Crawlee.
13,923Members
Resources
Recent Announcements
Similar Threads
Was this page helpful?
