liamk-ultra•2mo ago

Memory access problem on MacOS

I'm deploying on Linux, but my development system is MacOS. I discovered an issue that appears with concurrency > 2. I have plenty of memory but that's not actually the problem;, but in any case it should probably fail gracefully. The Snapshotter tries to access child process memory via psutil.Process.memory_full_info(), which requires elevated permissions on macOS and fails with

psutil/_psosx.py", line 356, in wrapper

    raise AccessDenied(self.pid, self._name)

psutil/_psosx.py", line 356, in wrapper

    raise AccessDenied(self.pid, self._name)

I came up with a monkey patch workaround, so this is not a pressing issue. Just thought I'd mention it in case anyone else encountered it, or a fix could be incorporated in the code.

Solution:

I think it is a bug introduced when improving memory usage estimation on Linux. Unfortunately we have tests only for Windows and Linux in CI, so macOS support is kind of fragile. Thanks for reporting it I will take a look into that https://github.com/apify/crawlee-python/issues/1329...

GitHub

Memory estimation problem on MacOS · Issue #1329 · apify/crawlee-...

It seems that fix: Use PSS instead of RSS to estimate children process memory usage on Linux introduced bug for MacOS as the memory_full_info might not be available there without some elevated priv...

Jump to solution

5 Replies

luigi.ruocco•2mo ago

I'm not sure I understand fully what are you trying to do or what caused the error. Can you give me more information about your workflow ?

Solution

Josef•2mo ago

GitHub

Memory estimation problem on MacOS · Issue #1329 · apify/crawlee-...

liamk-ultraOP•2mo ago

Wow! Thanks, Josef, that was fast. I've been using 0.6.12b34. Is there a beta tag that I should use to get this change?

Mantisus•2mo ago

@liamk-ultra The 0.6.12b43 release already includes this PR

liamk-ultraOP•2mo ago

Thanks. Just tested this and it works.

Memory access problem on MacOS

Did you find this page helpful?