大网数据广告活动
服务器运维问答

rsync hangs on one big file

· 52CE 我爱测

Each night I copy several virtual disks with rsync from one Linux Debian computer to another Linux Debian. Most files are raw images with "holes" into them: some parts were never written to, so remain unallocated on disk. rsync h…

问题原文

原帖链接 · Gregory MOUSSAT · CC BY-SA 3.0

Each night I copy several virtual disks with rsync from one Linux Debian computer to another Linux Debian.
Most files are raw images with "holes" into them: some parts were never written to, so remain unallocated on disk.

rsync hangs on one file, always the same. The hang occurs after 50 Gb transfered, and each time. I'm not sure if this is always at the exact same point, but ls -sh displays 50 Gb.
This is a 800 Gb file containing 151 Gb (so 649 Gb are unallocated). Some other virtual disks have similar figures, and rsync works well on them.

I have the exact same behaviour if I use rsync to update the file locally, without any network involvement (with --no-whole-file, this is a requirement, see later).

Once rsync is stalled, it uses one CPU core to 100% and zero disk activity on the receiving side (this is a pull request, so rsync is ran from this side) and zero CPU and zero disk on the sending side.
I let it ran during several hours.
Ctrl+c immediately stops rsync.
When ran to copy locally, once stalled I also have one CPU core to 100% and zero disk activity.

The only exception I found is when I rsync this file to a new location (ie the destination file does not exist). So I suspect the problem is related to the comparison between old and new data.

I use --inplace to limit writes to the destination disk, because snapshots are then used after each backup. So this option is a requirement, and rsync too, except if I found a tool able to update only changed parts of these files.

已采纳解答

原帖链接 · Bertrand SCHITS · CC BY-SA 3.0

rsync is known to have this kind of problem on huge files because of an inefficient algorithm for the internal hashes buffer.

You have to use the --block-size option with a big value. But current versions doesn't allow to use more than 128 kB. Some webpages tell to use --block-size=10485760 --protocol=29 but in my case it is refused by rsync.

  • use version 29 or older
  • or recompile after modification of the MAX_BLOCK_SIZE constant

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2016-03-09)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。