大网数据广告活动
服务器运维问答

Why is MemAvailable a lot less than MemFree+Buffers+Cached?

· 52CE 我爱测

I'm running a Linux workstation without swap and I have installed earlyoom daemon to automatically kill some processes if I'm running out of RAM. The earlyoom works by monitoring kernel MemAvailable value and if the available mem…

问题原文

原帖链接 · Mikko Rantalainen · CC BY-SA 4.0

I'm running a Linux workstation without swap and I have installed earlyoom daemon to automatically kill some processes if I'm running out of RAM. The earlyoom works by monitoring kernel MemAvailable value and if the available memory gets low enough, it kills less important processes.

This has worked fine for a long time but suddenly I'm now running into situation where MemAvailable is suddenly really low compared to the rest of the system. For example:

$ grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|Cached):' /proc/meminfo 
MemTotal:       32362500 kB
MemFree:         5983300 kB
MemAvailable:    2141000 kB
Buffers:          665208 kB
Cached:          4228632 kB

Note how MemAvailable is much lower than MemFree+Buffers+Cached.

Are there any tools I can run to further investigate why this happens? I feel that the system performance is a bit worse than normally and I had to stop the earlyoom service because its logic will not work unless MemAvailable is stable (that is, it correctly describes the available memory to user mode processes).

According to https://superuser.com/a/980821/100154 MemAvailable is an estimate of how much memory is available for starting new applications, without swapping. As I have no swap, what is this supposed to mean? Is this supposed to mean the amount of memory a new process can acquire before OOM Killer is triggered (because that would logically hit "the swap is full" situation)?

I had assumed that MemAvailable >= MemFree would be always true. Not here.

Additional info:

Searching around the internet suggests that the cause may be open files that are not backed by the filesystem and as a result, cannot be freed from the memory. The command sudo lsof | wc -l outputs 653100 so I definitely cannot manually go through that list.

The top of the sudo slabtop says

 Active / Total Objects (% used)    : 10323895 / 10898372 (94.7%)
 Active / Total Slabs (% used)      : 404046 / 404046 (100.0%)
 Active / Total Caches (% used)     : 104 / 136 (76.5%)
 Active / Total Size (% used)       : 6213407.66K / 6293208.07K (98.7%)
 Minimum / Average / Maximum Object : 0.01K / 0.58K / 23.88K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
4593690 4593656  99%    1.06K 153123       30   4899936K ext4_inode_cache
3833235 3828157  99%    0.19K 182535       21    730140K dentry
860224 551785  64%    0.06K  13441       64     53764K kmalloc-64
515688 510872  99%    0.66K  21487       24    343792K proc_inode_cache
168140 123577  73%    0.20K   8407       20     33628K vm_area_struct
136832 108023  78%    0.06K   2138       64      8552K pid
...

which looks normal to me.

Creating a rough summary of lsof

$ sudo lsof | awk '{ print $2 }' | sort | uniq -c | sort -h | tail
   6516 1118
   7194 2603
   7884 18727
   8673 19951
  25193 28026
  29637 31798
  38631 15482
  41067 3684
  46800 3626
  75744 17776

points to me PID 17776 which is a VirtualBox instance. (Other processes with lots of open files are Chrome, Opera and Thunderbird.) So I wouldn't be overly surprised to later figure out that the major cause of this problem is VirtualBox because that's the only thing that really messes with the kernel.

However, the problem does not go away even if I shutdown virtualbox and kill Chrome, Opera and Thunderbird.

已采纳解答

原帖链接 · Mikko Rantalainen · CC BY-SA 4.0

With modern Linux kernels the Cached value contains disk cache and all shared memory. As only disk cache can be dropped, all of Cached cannot be considered as available memory. I asked about this on the linux-mm mailing list and the general consensus was that the way Cached is calculated cannot be changed because of backwards compatibility, even though the current computation is a bit problematic for the label "cached". The traditional cached value should be computed as (Cached - Shmem) with modern kernels.

Nowadays you should just trust MemAvailable to describe the actually available memory in practice. And if you want more detailed breakdown, you should probably get data from /proc/meminfo and compute Free + Inactive(file) + KReclaimable. That computation differs from MemAvailable in a way that freeing inactive file cache will require IO. Also note that if kernel were to actually free all of KReclaimable, the overall throughput would probably suffer quite a bit, too. The MemAvailable is a heuristic that tries to guess the available RAM that can be used on top of the current workload without seriously hurting the throughput of the current workload. This is obviously different from theoretical max memory available without crashing the system.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2018-11-15)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。