Scaling Playwright Crawler on GCP Cloud Run
Yo everyone. Starting to try and hit my crawler with some real load.
Playwright setup in GCP Cloud Run... do you think a tonne of info logs I haven't cleaned up could be impacting performance? If so that's my lowest hanging fruit
Otherwise should I skip to looking at parallelization? What is performance like using lots of average/small/cheap instances versus overprovisioning a single cloud run instance and smacking it with load?
Not looked at the internals of crawlee too deeply so hoping the community can save me some time here
Playwright setup in GCP Cloud Run... do you think a tonne of info logs I haven't cleaned up could be impacting performance? If so that's my lowest hanging fruit
Otherwise should I skip to looking at parallelization? What is performance like using lots of average/small/cheap instances versus overprovisioning a single cloud run instance and smacking it with load?
Not looked at the internals of crawlee too deeply so hoping the community can save me some time here
