大网数据广告活动
服务器运维问答

NFS server as a client of itself performs very poorly

· 52CE 我爱测

I have set up an NFSv4 server. I am sharing files with Linux machines in a VM farm and Windows 7, 8, and 10 machines, so I need to share files with both NFS and Samba1. In the past, I have done this by configuring an NFS machine …

问题原文

原帖链接 · rg6 · CC BY-SA 4.0

I have set up an NFSv4 server. I am sharing files with Linux machines in a VM farm and Windows 7, 8, and 10 machines, so I need to share files with both NFS and Samba1. In the past, I have done this by configuring an NFS machine with direct access to the underlying ext4 file system and configuring a separate Samba server that mounts the NFS exports and shares them2.

I would like to unify the file sharing onto one server. Critically, I need to be sure that NFS and Samba are not stepping on each other's files, blissfully ignoring locks. As a result, I must continue using Samba on top of NFS and not in parallel with NFS.

This seems trivial. The NFS server should simply mount its own share, just as the Samba server would have done, and then that same machine must run Samba and export that mounted share. The only different would seem to be that traffic need not travel over the network between two machines. I would expect better performance.

Instead, the NFS server saturates the CPU on the machine while giving only 2.5 MB/s performance!

top - 11:34:57 up 15 days,  1:18,  1 user,  load average: 32.09, 8.07, 2.74
Tasks: 101 total,  13 running,  88 sleeping,   0 stopped,   0 zombie
%Cpu(s):  5.6 us, 93.7 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
KiB Mem :   899776 total,   124840 free,   184624 used,   590312 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   261184 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
11058 root      20   0       0      0      0 R 11.5  0.0   5:53.84 nfsd
11061 root      20   0       0      0      0 R 11.5  0.0   5:53.88 nfsd
11057 root      20   0       0      0      0 R 11.2  0.0   5:53.72 nfsd
11059 root      20   0       0      0      0 R 11.2  0.0   5:53.78 nfsd
11060 root      20   0       0      0      0 R 11.2  0.0   5:53.80 nfsd
11062 root      20   0       0      0      0 R 11.2  0.0   5:53.86 nfsd
11064 root      20   0       0      0      0 R 11.2  0.0   9:23.40 nfsd
11063 root      20   0       0      0      0 R 10.9  0.0   6:00.00 nfsd

The only other sign I know of that something is wrong is that countless lines get written to /var/log/messages that look something more or less like that.

Apr 25 11:35:01 nas kernel: nfsv4 compound op #4/4: 9 (OP_GETATTR)
Apr 25 11:35:01 nas kernel: nfsd: fh_verify(36: 01070001 00060001 00000000 9509b6fe 7a49ad56 5a4d5dbc)
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dbf606a080 opcnt 4 #4: 9: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound returned 0
Apr 25 11:35:01 nas kernel: --> nfsd4_store_cache_entry slot ffff88db7bbae000
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dac5072080 opcnt 4 #2: 22: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound op #3/4: 38 (OP_WRITE)
Apr 25 11:35:01 nas kernel: nfsd: write complete host_err=131072
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dac5072080 opcnt 4 #3: 38: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound op #4/4: 9 (OP_GETATTR)
Apr 25 11:35:01 nas kernel: nfsd: fh_verify(36: 01070001 00060001 00000000 9509b6fe 7a49ad56 5a4d5dbc)
Apr 25 11:35:01 nas kernel: nfsd_dispatch: vers 4 proc 1
Apr 25 11:35:01 nas kernel: nfsv4 compound op #1/4: 53 (OP_SEQUENCE)
Apr 25 11:35:01 nas kernel: __find_in_sessionid_hashtbl: 1555688119:2835816624:106:0
Apr 25 11:35:01 nas kernel: nfsd4_sequence: slotid 5
Apr 25 11:35:01 nas kernel: check_slot_seqid enter. seqid 9615 slot_seqid 9614
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dbf606a080 opcnt 4 #1: 53: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound op #2/4: 22 (OP_PUTFH)
Apr 25 11:35:01 nas kernel: nfsd: fh_verify(36: 01070001 00060001 00000000 9509b6fe 7a49ad56 5a4d5dbc)
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88db7b529080 opcnt 4 #4: 9: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound returned 0
Apr 25 11:35:01 nas kernel: --> nfsd4_store_cache_entry slot ffff88db7bbaf000
Apr 25 11:35:01 nas kernel: nfsd_dispatch: vers 4 proc 1
Apr 25 11:35:01 nas kernel: nfsv4 compound op #1/4: 53 (OP_SEQUENCE)
Apr 25 11:35:01 nas kernel: __find_in_sessionid_hashtbl: 1555688119:2835816624:106:0
Apr 25 11:35:01 nas kernel: nfsd4_sequence: slotid 1
Apr 25 11:35:01 nas kernel: check_slot_seqid enter. seqid 9626 slot_seqid 9625
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88db7b529080 opcnt 4 #1: 53: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound op #2/4: 22 (OP_PUTFH)
Apr 25 11:35:01 nas kernel: nfsd: fh_verify(36: 01070001 00060001 00000000 9509b6fe 7a49ad56 5a4d5dbc)
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dbf606f080 opcnt 4 #4: 9: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound returned 0
Apr 25 11:35:01 nas kernel: --> nfsd4_store_cache_entry slot ffff88db291eb000
Apr 25 11:35:01 nas kernel: nfsd_dispatch: vers 4 proc 1
Apr 25 11:35:01 nas kernel: nfsv4 compound op #1/4: 53 (OP_SEQUENCE)
Apr 25 11:35:01 nas kernel: __find_in_sessionid_hashtbl: 1555688119:2835816624:106:0
Apr 25 11:35:01 nas kernel: nfsd4_sequence: slotid 2
Apr 25 11:35:01 nas kernel: check_slot_seqid enter. seqid 9614 slot_seqid 9613
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dbf606f080 opcnt 4 #1: 53: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound op #2/4: 22 (OP_PUTFH)
Apr 25 11:35:01 nas kernel: nfsd: fh_verify(36: 01070001 00060001 00000000 9509b6fe 7a49ad56 5a4d5dbc)
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dac507e080 opcnt 4 #2: 22: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound op #3/4: 38 (OP_WRITE)
Apr 25 11:35:01 nas kernel: nfsd: write complete host_err=131072
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dac507e080 opcnt 4 #3: 38: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound op #4/4: 9 (OP_GETATTR)
Apr 25 11:35:01 nas kernel: nfsd: fh_verify(36: 01070001 00060001 00000000 9509b6fe 7a49ad56 5a4d5dbc)
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dac5072080 opcnt 4 #4: 9: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound returned 0
Apr 25 11:35:01 nas kernel: --> nfsd4_store_cache_entry slot ffff88dbf65cc000
Apr 25 11:35:01 nas kernel: nfsd_dispatch: vers 4 proc 1
Apr 25 11:35:01 nas kernel: nfsv4 compound op #1/4: 53 (OP_SEQUENCE)
Apr 25 11:35:01 nas kernel: __find_in_sessionid_hashtbl: 1555688119:2835816624:106:0
Apr 25 11:35:01 nas kernel: nfsd4_sequence: slotid 4
Apr 25 11:35:01 nas kernel: check_slot_seqid enter. seqid 9584 slot_seqid 9583
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dac5072080 opcnt 4 #1: 53: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound op #2/4: 22 (OP_PUTFH)
Apr 25 11:35:01 nas kernel: nfsd: fh_verify(36: 01070001 00060001 00000000 9509b6fe 7a49ad56 5a4d5dbc)
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dbf606a080 opcnt 4 #2: 22: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound op #3/4: 38 (OP_WRITE)
Apr 25 11:35:01 nas kernel: nfsd: write complete host_err=131072
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dbf606a080 opcnt 4 #3: 38: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound op #4/4: 9 (OP_GETATTR)
Apr 25 11:35:01 nas kernel: nfsd: fh_verify(36: 01070001 00060001 00000000 9509b6fe 7a49ad56 5a4d5dbc)

I am not mounting the NFS shares locally via 127.0.0.1. The local NFS mount uses the private IP address of the machine itself. I could change that, but it would require additional complexity in the NFS exports definition, and I would actually prefer that this machine sees itself exactly the same way as the other machines that will be mounting the NFS shares3.

Footnotes

  1. I know of ways of using NFS clients on Windows and Samba clients on Linux. Such options are not workable in this case.
  2. I am aware of admonishments about re-exporting NFS shares with Samba, but I and many other sysadmins I know regularly do this without problems. The issue seems to have arisen here rather because the NFS server is mounting its own share. Nonetheless, I am curious about the exact reasons for such warnings, so if you know please share.
  3. If anybody has some reason why this entire architecture is fundamentally flawed, I am open to hearing that. I have years of experience serving Samba on top of NFS using two machines, so I supposed that I could cut out the Samba machine and move Samba to the NFS server machine with an NFS self-mount.

已采纳解答

原帖链接 · Spooler · CC BY-SA 4.0

It's very likely that your workload is dying a scheduling death. You're asking for a ton on context switches on the same system when exporting a filesystem back to itself and then exporting that export again.

You may see that splitting this workload out across more than one kernel (kernels that do not share the same CPU core) results in a performance increase, despite the involvement of a network link between those systems.

This could easily be accomplished with a collocated VM set; one running NFS, one running Samba. Collocating these on the same hypervisor should eliminate most network latencies, and should work well provided that these VMs do not share the same CPU cores.

There are also userspace implementations for NFS - NFS Ganesha being a very common one. However, Samba is also going to be responsible for a non-trivial amount of this repetitive userspace-kernel switching. NFS Ganesha may be worth testing if these workloads absolutely must share a kernel for some reason.

In short, reducing context switching within the same system should help significantly with this.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2019-04-25)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。