大网数据广告活动
服务器运维问答

Backing up large files using rsync to external drives (NTFS) taking into consideration performance & speed

· 52CE 我爱测

Currently files are backed up from one external hard drive to another as a means to provide the ability to restore from a secondary device should the primary external hard drive become unavailable e.g. hardware failure. The curre…

问题原文

原帖链接 · Motivated · CC BY-SA 3.0

Currently files are backed up from one external hard drive to another as a means to provide the ability to restore from a secondary device should the primary external hard drive become unavailable e.g. hardware failure. The current method is predicated on the use of Microsoft Windows.

All backup is to be performed from Linux devices running Ubuntu 14.04 going forward.

All external hard drives are encrypted using TrueCrypt and have been formatted with NTFS since they are also used on other operating systems other than Linux such as Microsoft Windows.

Files are also created on Microsoft Windows and stored on these devices.

File sizes vary from as little as 100KB to as large as 10GB. There are approximately 1 million files and this continues to grow on a regular basis.

Requirements

  • Performance on external hard drives. All current drives are USB 2
  • Speed of transfer
  • Compute thresholds or considerations. The devices that currently perform the backups are either i3, i5, i7 with a minimum of 8GB of RAM
  • Preserving permissions and the like
  • Deep paths. Some paths traverse as deep as 20 or 30 hierarchies
  • Minimal impact on the lifespan of the external hard drives
  • Avoidance of changing filesystems if possible
  • Whole copies are fine and deltas or differentials are not required
  • Checksums to ensure that files have been successfully committed
  • Logs
  • The ability to resume transfers in the event of a failure

Now there have been mention of using switches or options such as inplace however there is very little information available on additional considerations for encrypted drives that leverage NTFS filesystems.

EDIT

The current attempted command as an example is;

rsync -vrlptg /media/drive1 /media/drive2

What other switches/options must be included to support the requirements?

已采纳解答

原帖链接 · shodanshok · CC BY-SA 3.0

Considering you are using a slow USB 2.0 protocol, it is critical to transfer as little data as possible.

Rsync is your best friend here. However, NTFS is not making you favor. Using something as EXT4 enable you to leaverage hardlinks to only store the changed files. Take a look at rsnapshot for further information

If you can't change your filesystem, you can also have a look at attic-backup, so you can have deduplicated backup without resorting to hardlinks.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2016-01-07)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。