How to combine the scraping results for Crawlee Playwright actor?
Hello folks! I'm building an actor with Crawlee and playwright for bbb scraping. I have completed the "detail" content parsing, and I made two booleans inputs to let users decide whether to scrape reviews, complaints or not.
Then I realized if they choose both, I'll need to send two more requests and I don't know how to use the router to combine the results(the reviews, complaints and detail results) before using
Then I realized if they choose both, I'll need to send two more requests and I don't know how to use the router to combine the results(the reviews, complaints and detail results) before using
Dataset.pushDataDataset.pushData.
Solution
You need a way to persist state per entity (e.g., one business from BBB) across multiple requests, and only pushData() once all requested pieces (detail + reviews + complaints) are collected.
In Crawlee, you do this by:
In Crawlee, you do this by:
