Trying to optimize autoscale options

Hello,

I am running my scraper on an AWS 8gb cpu, 16gb memory ecs.

    maxConcurrency: 200,
    maxRequestsPerCrawl: 500,
    maxRequestRetries: 2,
    requestHandlerTimeoutSecs: 185,

    maxConcurrency: 200,
    maxRequestsPerCrawl: 500,
    maxRequestRetries: 2,
    requestHandlerTimeoutSecs: 185,

Right now the avg cpu and mem are both like 88%. Is there anything I can do here to optimize more?

I also have

CRAWLEE_AVAILABLE_MEMORY_RATIO=.8

CRAWLEE_AVAILABLE_MEMORY_RATIO=.8

Hi @bmax, this is a case-by-case thing. It highly depends on scraped sites, whether you are using a browser, browser settings,...

technological-jadeOP•4/11/24, 4:37 AM

@vojtechmaslan any guide lines?

MEE6•4/11/24, 4:37 AM

@bmax just advanced to level 5! Thanks for your contributions!

technological-jadeOP•4/11/24, 4:38 AM

also cpu seems to hit 99% no matter what

technological-jadeOP•4/11/24, 4:58 AM

what does eventLoop overloaded mean?

technological-jadeOP•4/11/24, 5:14 AM

or how come currentConccurency is 18 when I have maxConcurrency at 200 and there are plenty of request?

Oleg V.•4/29/24, 2:43 PM

Oleg V.•4/29/24, 2:44 PM