4 replies

How to combine the scraping results for Crawlee Playwright actor?

Hello folks! I'm building an actor with Crawlee and playwright for bbb scraping. I have completed the "detail" content parsing, and I made two booleans inputs to let users decide whether to scrape reviews, complaints or not.
Then I realized if they choose both, I'll need to send two more requests and I don't know how to use the router to combine the results(the reviews, complaints and detail results) before using Dataset.pushDataDataset.pushData.

Solution

You need a way to persist state per entity (e.g., one business from BBB) across multiple requests, and only pushData() once all requested pieces (detail + reviews + complaints) are collected.

In Crawlee, you do this by:

1. Storing a “partial result” in request.userData.
2. Passing it along to subsequent requests.
3. Once all needed data are gathered, pushing it to the dataset.

1. Storing a “partial result” in request.userData.
2. Passing it along to subsequent requests.
3. Once all needed data are gathered, pushing it to the dataset.

Jump to solution

Apify & Crawlee•9mo ago•

4 replies

Jack Wenyoung

How to combine the scraping results for Crawlee Playwright actor?

Solution

1. Storing a “partial result” in request.userData.
2. Passing it along to subsequent requests.
3. Once all needed data are gathered, pushing it to the dataset.

1. Storing a “partial result” in request.userData.
2. Passing it along to subsequent requests.
3. Once all needed data are gathered, pushing it to the dataset.

Jump to solution

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,851 Members

Join

How to combine the scraping results for Crawlee Playwright actor?

How to combine the scraping results for Crawlee Playwright actor?

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads