Memory access problem on MacOS
I'm deploying on Linux, but my development system is MacOS.
I discovered an issue that appears with concurrency > 2.
I have plenty of memory but that's not actually the problem;, but in any case it should probably fail gracefully.
The Snapshotter tries to access child process memory via
which requires elevated permissions on macOS and fails with
I came up with a monkey patch workaround, so this is not a pressing issue. Just thought I'd mention it in case anyone else encountered it, or a fix could be incorporated in the code.
I discovered an issue that appears with concurrency > 2.
I have plenty of memory but that's not actually the problem;, but in any case it should probably fail gracefully.
The Snapshotter tries to access child process memory via
psutil.Process.memory_full_info()which requires elevated permissions on macOS and fails with
I came up with a monkey patch workaround, so this is not a pressing issue. Just thought I'd mention it in case anyone else encountered it, or a fix could be incorporated in the code.
Solution
I think it is a bug introduced when improving memory usage estimation on Linux. Unfortunately we have tests only for Windows and Linux in CI, so macOS support is kind of fragile. Thanks for reporting it I will take a look into that
https://github.com/apify/crawlee-python/issues/1329
https://github.com/apify/crawlee-python/issues/1329
GitHub
It seems that fix: Use PSS instead of RSS to estimate children process memory usage on Linux introduced bug for MacOS as the memory_full_info might not be available there without some elevated priv...
This is the official developer community of Apify and Crawlee.
14,091Members
Resources
Recent Announcements
Similar Threads
Was this page helpful?
