4 replies

How to combine the scraping results for Crawlee Playwright actor?

Hello folks! I'm building an actor with Crawlee and playwright for bbb scraping. I have completed the "detail" content parsing, and I made two booleans inputs to let users decide whether to scrape reviews, complaints or not.
Then I realized if they choose both, I'll need to send two more requests and I don't know how to use the router to combine the results(the reviews, complaints and detail results) before using

Dataset.pushData

Dataset.pushData

Dataset.pushData

Dataset.pushData.

Solution

You need a way to persist state per entity (e.g., one business from BBB) across multiple requests, and only pushData() once all requested pieces (detail + reviews + complaints) are collected.

In Crawlee, you do this by:

1. Storing a “partial result” in request.userData.
2. Passing it along to subsequent requests.
3. Once all needed data are gathered, pushing it to the dataset.

1. Storing a “partial result” in request.userData.
2. Passing it along to subsequent requests.
3. Once all needed data are gathered, pushing it to the dataset.

1. Storing a “partial result” in request.userData.
2. Passing it along to subsequent requests.
3. Once all needed data are gathered, pushing it to the dataset.

1. Storing a “partial result” in request.userData.
2. Passing it along to subsequent requests.
3. Once all needed data are gathered, pushing it to the dataset.

Jump to solution

How to combine the scraping results for Crawlee Playwright actor?

Similar Threads

Similar Threads

Similar Threads