大网数据广告活动
服务器运维问答

Does rsync preserve hardlinks on the destination when source files are identical but separate?

· 52CE 我爱测

I am running a (Linux based) rsync server for software distribution. A (Windows based) source repository server which is outside my control pushes software packages to it via rsync, and about a hundred satellite servers worldwide…

问题原文

原帖链接 · Tilman Schmidt · CC BY-SA 4.0

I am running a (Linux based) rsync server for software distribution. A (Windows based) source repository server which is outside my control pushes software packages to it via rsync, and about a hundred satellite servers worldwide pull from it, also via rsync.

The source repository contains many big duplicate files. I want to reduce disk space and bandwidth consumption on the satellite servers by replacing those duplicates by hardlinks. The administrator of the source repository is unwilling or unable to do so at the source, so I'm trying to do it after the fact on the distribution server. I have created a simple bash script based on the fdupes command which finds groups of duplicates and replaces them with hardlinks to a single file. The rsync transfers to the satellite servers preserve these hardlinks as desired thanks to the -H option. The transfer from the source repository however produces inconsistent results. Sometimes the deduplication is preserved. Sometimes the source server retransmits all of the files of a deduplicated group and the deduplication is broken even though the source files did not change.

Hence my question: What is the official behaviour of rsync in case it is asked to sync two identical but separate files and the files do already exist in the destination with the correct content, but as hardlinks to the same file? What is the exact criteria for retransmitting a file? Is there a way to ensure that the hardlink in the destination is preserved in that situation even though the hardlink does not exist in the source?

已采纳解答

原帖链接 · Tilman Schmidt · CC BY-SA 4.0

tl;dr: To preserve file level deduplication via hard links at the destination, run rsync with the --checksum option.

Full answer, according to a series of experiments I did:

If two files are not hardlinked at the source, rsync will sync each of them individually to the destination. It does not care whether the files happen to be hardlinked at the destination. If one of the files (or both of them) ends up being retransmitted, the hard link at the destination will be broken, otherwise it will be untouched. That is, even with the --hard-links option, rsync will not break a hardlink at the destination just because the files are not hardlinked at the source.

The criteria for retransmitting a file depend on the --checksum (-c) and --ignore-times (-I) options.

  • If the option --checksum is given, only files that differ in size or checksum between source and destination are retransmitted. Consequently, if the file content hasn't changed then a hard link at the destination will be preserved even if it doesn't exist at the source.
  • If the option --ignore-times is given, all files are retransmitted, breaking any hard link at the destination that doesn't exist at the source.
  • If neither of these two options is given, rsync will use the modification timestamps of the source and destination files for its decision. In that case, if the timestamps of the two source files differ, a hard link at the destination will always be broken because only one of the two timestamps can match.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2020-12-10)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。