Page Navigation is Slow when Running in Apify

I noticed my crawler ran much slower when running in Apify than in my laptop.

In my local (Mac M1 Pro 16G, headless=false): page navigation takes ~ 3-8 seconds;
in Apify (2048MB, headless=true): page navigation takes ~ 10~20 seconds.

is the difference purely caused by hardware? or is there anything I can improve? thanks!

Xxli I noticed my crawler ran much slower when running in Apify than in my laptop. I...

HonzaS•11/9/22, 6:49 PM

You can try to increase memory, it will increase cpu power as well. That way you will find out if it is hardware dependent.

HonzaS•11/9/22, 6:51 PM

it could also be because of slow proxy connection, you can increase concurrency to speed up whole scraping process

worthy-roseOP•11/9/22, 10:58 PM

thanks

Lukas Krivka•11/11/22, 10:19 AM

You kinda answered yourself by comparing your laptop specs with the actor. You need to at least match the memory.

worthy-roseOP•11/11/22, 4:42 PM

I was thinking the page navigation was mostly affected by bandwidth (which Apify should be better than my home network), but it turned all all resources including memory/cpu mattered.

worthy-roseOP•11/11/22, 4:48 PM

btw, is there any difference Apify manages free v.s. paid plans? say for example, if I run my actor with 2G memory, will it be more ensured to have that 2G computing power in paid than in free plan?

MEE6•11/11/22, 4:48 PM

@xli just advanced to level 1! Thanks for your contributions!

Lukas Krivka•11/14/22, 12:20 PM

No, paid plan just allows you to even use more GB, otherwise all specs are the same

Lukas Krivka•11/14/22, 12:21 PM

But you get the memory/CPU you ask for, it is using Docker to assign these resources