Apify & Crawlee•4y ago

crawlee eating memory like hell

It is eating 3 GB after running just for 2 days

ill-bronzeOP•11/1/22, 12:50 AM

i think this is the issue

ill-bronzeOP•11/1/22, 12:50 AM

ill-bronzeOP•11/1/22, 12:50 AM

I am creating new url every time and it is caching them

ill-bronzeOP•11/1/22, 12:51 AM

if i don't do that then it won't scrap duplicate url but i want to scrap time to time

ill-bronzeOP•11/1/22, 12:53 AM

i asked fix for this issue but no response last time

ill-bronzeOP•11/1/22, 12:53 AM

would be very helpful if you can help me out this time

excited-coffee•11/1/22, 11:16 AM

I'm quite new to this stuff too. But I did a load of digging around and stuff.

I can't confim if this all works yet as I'm still experimenting. But I've moved the adding of the urls into a RequestQueueRequestQueue. This is something you can pass into the crawler when you call the constructor on it. You can call open on it statically and then also assign it an id. This will now put it in storage under the ID and not default. It looks like so

const id = `${parsedUrl.host}_${Date.now()}`;
const requestQueue = await RequestQueue.open(id);

const id = `${parsedUrl.host}_${Date.now()}`;
const requestQueue = await RequestQueue.open(id);

Pass that to the crawler. Now I can run multiple queue of the same domain multiple times. RequestQueueRequestQueue also has a drop method which is suppose to remove it from the storage for you. (however I'm not entirely certain I've got this working completely yet)

I'm calling my stored RequestQueueRequestQueue in a requestQueuerequestQueue variable. Once the crawler has finsihed running. I call requestQueue.drop()requestQueue.drop() hopefully this is tidying up the storage but I'm currently still not totally sure. I hope this helps you out a bit!

await crawler.run();
// removes it from storage
await requestQueue.drop();

await crawler.run();
// removes it from storage
await requestQueue.drop();

Lukas Krivka•11/2/22, 8:26 PM

@CTK WARRIOR This is not enough context to debug this. The code you shared looks fine, the request objects themselves are small unless you hold something big in userData

Cclean-aquamarine @CTK WARRIOR This is not enough context to debug this. The code you shared looks...

ill-bronzeOP•11/2/22, 8:28 PM

Yeah I am passing product info in userData

ill-bronzeOP•11/2/22, 8:29 PM

which contains name, url and few other details

ill-bronzeOP•11/2/22, 8:29 PM

i am creating new crawlee object for each turn

Lukas Krivka•11/2/22, 8:53 PM

I don't see Crawlee itself to be memory hungry. The sources of memory usage is usually browsers/cheerio parsing or user data

Cclean-aquamarine I don't see Crawlee itself to be memory hungry. The sources of memory usage is u...

ill-bronzeOP•11/4/22, 12:34 AM

How can I like clean the memory?

Eexcited-coffee I'm quite new to this stuff too. But I did a load of digging around and stuff. ...

ill-bronzeOP•11/6/22, 9:03 PM

tried this but no change

ill-bronzeOP•11/6/22, 9:03 PM

after running for few hours, it starts to eat alot of ram

Lukas Krivka•11/7/22, 5:26 PM

You need to delete the references to objects you don't need. There is nothing leaking in Crawlee specifically.

Cclean-aquamarine You need to delete the references to objects you don't need. There is nothing le...

ill-bronzeOP•11/7/22, 7:31 PM

here is my full code

target.js5.74KB

ill-bronzeOP•11/10/22, 11:49 PM

@Lukas Krivka can you taka a glance at my code and let me know if anything is wrong

ill-bronzeOP•11/10/22, 11:49 PM

sorry for ping

Lukas Krivka•11/11/22, 10:23 AM

So I assume the memory is not getting clearer after each cron?

Cclean-aquamarine So I assume the memory is not getting clearer after each cron?

ill-bronzeOP•11/12/22, 2:35 AM

yeah yeah

ill-bronzeOP•11/12/22, 2:35 AM

its 500 MB after 17 hours and after 2-3 days, its 1 GB

efficient-indigo•11/12/22, 7:39 AM

Perhaps you must use profiling tools to watch memory allocation and find memory leaks.
To help you detect memory leaks, you can use this fork of Memwatch (https://github.com/airbnb/node-memwatch). This module is useful because it can emit leak events if it sees the heap grow over 5 consecutive garbage collections.

efficient-indigo•11/12/22, 7:41 AM

Clinic.js (https://clinicjs.org/) is another tool to help diagnose and pinpoint Node.js performance issues.

ill-bronzeOP•11/14/22, 3:35 AM

No lead so far

ill-bronzeOP•11/14/22, 3:36 AM

The only option I have is using child process to run crawlee