大网数据广告活动
服务器运维问答

Slow performance copying large file over network (scp)

· 52CE 我爱测

I have a Linux box that I use as a file server. I have a monthly cron job that tars up the contents of the data drive and then copies it via scp to another machine for safe keeping. The resulting tarball is around 300GB in size, …

问题原文

原帖链接 · Jeff Loughlin · CC BY-SA 3.0

I have a Linux box that I use as a file server. I have a monthly cron job that tars up the contents of the data drive and then copies it via scp to another machine for safe keeping. The resulting tarball is around 300GB in size, and it normally takes about a day and a half to complete the copy (over an 802.11g Wi-Fi connection).

Today I noticed that my backup job hadn't completed yet, and it's been running for 3 days. Checking the destination machine, I could see that only about a third of the data has been copied so far, and it appears to be growing at a rate of < 300KB/sec.

Using iperf between the two machines, I can see that my network throughput is around 20Mbits/sec, which is about what I expect over an 802.11g connection.

Using dd if=srcfile of=/dev/null on the source machine, I can read about 45MB/sec from the source drive (an external USB drive).

using dd if=/dev/zero of=/destdrive/tmp.dat on the destination machine, I can write about 30MB/sec to the destination drive (internal SATA drive). Seems kind of slow for a SATA drive, but not unreasonably slow (and certainly not 300KB/sec slow).

So I seem to have ruled out network throughput and drive throughput at both ends, so where else can I look to find the source of the bottleneck?

社区高分解答

原帖链接 · Ijaz Ahmad · CC BY-SA 3.0

Why you are using scp for copying large files in the first place? scp has its own overhead (encryption, authenticity checking, etc).

You can use rsync (rsync is very well suited for transferring large files over ssh because it is able to continue transfers that were interrupted due to some reason. Since it uses hash functions to detect equal file blocks the continue feature is quite robust.) or some other tool.

Please see this post. Copying large files over network , faster

If you want to use scp anyway , then you should use traceroute and tcpdump and iftopto see the packets from source to destination. May be you find something unusual.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2016-11-23)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。