Parallel crawling using both Cheerio and Playwright?
🎭PlaywrightCrawler🚀RequestQueue👨💻Web-Scraping🍩CheerioCrawler
Can PlaywrightCrawler and CheerioCrawler be used in parallel? For example, can Playwright push a URL (got detail page URL by javascript rendering) to CheerioCrawler request queue for another immediate crawling (get the detailed page json data)? I noticed Request Queue V2, but the official examples seem aimed at speeding up crawling via process forking.
I am looking for a more lightweight approach where CheerioCrawler handles the remaining work without the two blocking each other.
Do you have any insights on this?
I am looking for a more lightweight approach where CheerioCrawler handles the remaining work without the two blocking each other.
Do you have any insights on this?
Solution
Thanks for your reply. I have used two crawler with promise.all to make the crawler be parallelized running.
The important thing is:
1. add a isSearchDone callback when create a new crawler (at the entrypoint) just like you pasted
2.add a custom request sink function and pass it to first cralwer, in the first cralwer router calling the sink function
push the new urls to cherrio crawler. It works perfect with my setup(playwright for js rendering and cheerio for lightweight fast SSR page scrapping) but haven't tested in a large scaled cralwers(I believe this is enough for most cases)
The important thing is:
1. add a isSearchDone callback when create a new crawler (at the entrypoint) just like you pasted
2.add a custom request sink function and pass it to first cralwer, in the first cralwer router calling the sink function
push the new urls to cherrio crawler. It works perfect with my setup(playwright for js rendering and cheerio for lightweight fast SSR page scrapping) but haven't tested in a large scaled cralwers(I believe this is enough for most cases)
