大网数据广告活动
服务器运维问答

Debug slow lan (ssh, nfs) file transfer

· 52CE 我爱测

I've got two linux boxes attached to a gigabit switch. They both have gigabit NICs, cables are cat7. Testing the network with iperf shows a fast connection but transferring files with rsync, scp, or nfs share is slow. I'm testing…

问题原文

原帖链接 · pinpox · CC BY-SA 3.0

I've got two linux boxes attached to a gigabit switch. They both have gigabit NICs, cables are cat7.

Testing the network with iperf shows a fast connection but transferring files with rsync, scp, or nfs share is slow.

I'm testing with one 1GB file.

iperf result:

Client connecting to odroid, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.26 port 58788 connected with 192.168.1.32 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   979 MBytes   821 Mbits/sec

The transfer speed with rsync, scp or nfs is all about ~13Mb/s

scp:

 scp bigfile odroid:/mnt/usb1/               [10:19:12]
bigfile                                        57%  590MB  12.2MB/s   00:35 ETA^CKilled by signal 2.

rsync:

 rsync --progress bigfile /mnt/usb1/
bigfile
     44,695,552   4%   12.15MB/s    0:01:11  ^C

nfs:

binaryplease➜~(master✗)» time cp bigfile /mnt/nfs/usb1/      
cp -i bigfile /mnt/nfs/usb1/  0.01s user 0.94s system 1% cpu 1:11.06 total

1024MB/71sec = 14,42 MB/s

Since the iperf test shows a fast network connection, I assumed a problem with the storage devices being slow, but that doesn't seem to be the case either:

Client, SSD, internal:

binaryplease➜~(master✗)» sudo hdparm -tT /dev/sda                    
/dev/sda:
 Timing cached reads:   20344 MB in  2.00 seconds = 10181.50 MB/sec
 Timing buffered disk reads: 1498 MB in  3.00 seconds = 498.98 MB/sec

binaryplease➜~(master✗)» dd if=/dev/zero of=test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 2.03861 s, 263 MB/s

binaryplease➜~(master✗)» dd if=test of=/dev/null iflag=direct bs=8M  [12:29:01]
64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 1.11392 s, 482 MB/s

Server, USB 3.0 Drive, external:

➜  ~ git:(master) ✗ sudo hdparm -tT /dev/sda   
/dev/sda:
 Timing cached reads:   1980 MB in  2.00 seconds = 991.66 MB/sec
 Timing buffered disk reads: 266 MB in  3.01 seconds =  88.27 MB/sec
➜  usb1   dd if=/dev/zero of=test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 6.53386 s, 82.2 MB/s
➜  usb1  dd if=test of=/dev/null iflag=direct bs=8M
64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 7.13567 s, 75.2 MB/s

OS on client (Linux arch):

Linux binaryplease-laptop 4.3.3-2-ARCH #1 SMP PREEMPT Wed Dec 23 20:09:18 CET 2015 x86_64 GNU/Linux

OS on server (Ubuntu server for odroid):

Linux odroid 3.10.92 #1 SMP PREEMPT Tue Nov 17 00:15:24 BRST 2015 armv7l armv7l armv7l GNU/Linux

On both systems neither the cpu or the ram is maxed out.

If I interpret the results correctly the write speed of the servers drive (82.2 MB/s) should be easily matched by the network. How is the file transfer so slow?

I hope the information provided is sufficient and someone can help me find the bottleneck.

Thanks.

已采纳解答

原帖链接 · Overmind · CC BY-SA 3.0

At the numbers you get, seems like something is not actually gigabit in your network.

Use ethtool or something similar to check the speed on each machine NIC.

Check the switch and make sure the ports are actually active at gigabit speed (check LEDs or configuration port status if the switch supports management).

Run iperf or an alternate between the clients.

Finally, check CPU usage - perhaps something is wrong there.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2016-01-22)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。