adverse-sapphire•3y ago

Split crawler and scraper into 2 container

I want to split crawler & scraper container, scraper container can call crawler container to get html. 2 container call by gRPC or JSON RPC. Purpose for this, I want 2 process is independent (crawler container can cache or store own data) Thank you!

4 Replies

fascinating-indigo•3y ago

This is very possible, but what have you already tried? We can help you with a problem you're facing, but not create an entire solution for you.

adverse-sapphireOP•3y ago

I need to run PupperteerCrawler in the gRPC server and scraper container to get data from Crawler vi gRPC protocal

Lukas Krivka•3y ago

This is not specific to Crawlee in any way. Just send the URLs to the seconds container

adverse-sapphireOP•3y ago

thank you

Split crawler and scraper into 2 container

Did you find this page helpful?