大网数据广告活动
服务器运维问答

How to monitor RAM consumption on Red Hat 7.9

· 52CE 我爱测

Red Hat 7.9 server with 512 GB RAM. We often have alerts about swap being full. Swap is often used 99%. Our server admin told us it is normal for linux to have swap used 100%. There is no way to check the real RAM consumption. Ou…

问题原文

原帖链接 · iostrym · CC BY-SA 4.0

Red Hat 7.9 server with 512 GB RAM.

We often have alerts about swap being full. Swap is often used 99%. Our server admin told us it is normal for linux to have swap used 100%. There is no way to check the real RAM consumption.

Ours is always around 99% on our server using SAR:

Memory & Swap
=============
                    kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
01/11/2024    9869583 517833049     98.13       977 285023790 319073740     58.60 340613912 162678784    614843
01/12/2024    4181004 523521628     99.21      1349 287757937 323767076     59.46 346046454 162168337   1115633
01/13/2024    2567787 525134845     99.51       844 285755715 327744180     60.19 352144911 157031086    654493
01/14/2024    2827695 524874937     99.46       844 285135562 328070493     60.25 352003644 156742082    742178
01/15/2024    3482087 524220545     99.34      1838 280133083 332837943     61.13 353676271 153986907    998373
01/16/2024    2152990 525549642     99.59       839 273578756 342252974     62.86 362157756 147013751   1136099
01/17/2024    4575639 523126993     99.13      2418 271393967 340334778     62.51 355987093 150884756   1033531
01/18/2024    2205445 525497187     99.58      2413 282078831 328216144     60.28 353148916 156770066    625021
01/19/2024    9451354 518251278     98.21      1542 293648210 305176716     56.05 352495156 150264497    680464

On the web I can see that it is not normal to have high swap usage. I also noticed that after a reboot, swap is very low during several days. When a process consumes all the RAM (mistake on our side), then swap usage increases to 100%, and then never decreases, even if the corresponding process is killed. This is why servers that are on for weeks have 100% swap used.

I was told to monitor the swap usage using sar (pswpin/s pswpout/s).

When swap usage is 100%, I could have no problem but when processes start being killed because of RAM issues, I can see high values for pswpin/s pswpout/s (sar -W).

During a week I can monitor this activity to check if I had RAM problems or not.

My problem is the following: How can I prevent the RAM problem from occurring? What can I use to check the % of RAM used (whereas it is always 99% in SAR...)? How to get the real value like in Windows OS? To be sure to kill the process that start taking all the RAM.

I would like to generate a warning when RAM usage is 80%.

I know that free -h could be used but I don't know how to interpret that. the same for "top".

For r instance, I compare sar output with free -h and top output and I don't see matching values... :-(

[XXXXX@YYYYYY ~]$ sar -r
Linux 3.10.0-1062.4.1.el7.x86_64 (XXXXXX)   02/02/2024  _x86_64_    (64 CPU)
10:30:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty

11:55:01 AM   7916396 519786232     98.50      4688 436720908 164442944     30.20 256757744 249531760      1548
12:00:01 PM   1106680 526595948     99.79      4688 436883120 173459020     31.86 263356476 249699552      4184
12:05:01 PM    742056 526960572     99.86      4688 436447380 173668428     31.90 264216776 249402376      5380
12:10:01 PM  11076780 516625848     97.90      4688 434763392 162944064     29.93 255501116 247835888      3732
12:15:01 PM   7891084 519811544     98.50      4688 434921220 165981024     30.48 258656448 247885164       600
Average:      2201518 525501110     99.58      5447 448667788 154099963     28.30 257069976 255388912      3431
[XXXXX@YYYYYY ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:           503G         81G        4.3G        1.3G        417G        419G
Swap:           15G         11G        4.4G
[XXXXX@YYYYYY ~]$ top
top - 12:22:09 up 9 days, 18:55, 48 users,  load average: 4.21, 4.69, 5.06
Tasks: 2645 total,   5 running, 2488 sleeping,   0 stopped, 152 zombie
%Cpu(s):  6.8 us,  3.4 sy,  0.0 ni, 89.8 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 52770262+total,  6888876 free, 83310656 used, 43750310+buff/cache
KiB Swap: 16777212 total,  4582892 free, 12194320 used. 44195168+avail Mem 

I don't know which values to check on top or free -h to know the real % of RAM consumption usage.

thanks a lot for helping

已采纳解答

原帖链接 · shodanshok · CC BY-SA 4.0

Your system does not seem to have any swap-related issue. However, as you mention high swap use and killed processes, you can have occasional low-memory issues.

By default, some swap is expected to be used even on system with plenty of free memory as the kernel proactively swap inactive memory to have more room for buffer/cache. You have 500+ GB of RAM and a 16 GB swap partition, so it is normal to fill it with inactive data.

If swap is causing you performance issues, you can limit or disable its use with the following:

  • sysctl vm.swappiness=0 means the system should not use any swap unless free memory is dangerously very low. To persists across reboot, you need to put vm.swappiness=0 into /etc/sysctl.conf file;

  • swapoff -a forcefully disable any swap. Please note that if you genuinely need some swap (ie: to absorb a memory consumption peak) your system could kill some process via OOM-killer. To persist across reboot, you need to remove the swap entry from /etc/fstab file;

  • disable disk swap and use zram to create a compressed in-RAM device. zram can be very useful but I would not use this solution unless really needed and applicable to your workload.

Some more info:

When a process consumes all the RAM (mistake on our side), then swap usage increases to 100%, and then never decreases, even if the corresponding process is killed

When a swapped process exists, swap should be released. If swap is constantly filled it probably means that another non-swapped process was killed, rather than the swapped one. You can use dmesg to check for kernel OOM.

When swap usage is 100%, I could have no problem but when processes start being killed because of RAM issues, I can see high values for pswpin/s pswpout/s (sar -W).

This means the kernel is (unsuccessfully) trying to free some memory to prevent OOM.

I would like to generate a warning when RAM usage is 80%.

sar does not have alerting capability as far I know. You need to parse free output and/or use a network monitor tool as zabbix, netdata, etc.

I know that free -h could be used but I don't know how to interpret that. the same for "top".

Well, you should really read the relevant man pages. In short, you can check the used or available columns reported by free.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2024-02-02)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。