大网数据广告活动
服务器运维问答

How to make rsync of ~2M files from remote server performant for regular backups

· 52CE 我爱测

We have a large amount of files on a remote server that I'd like to setup regular backups to a local system for extra redundancy. Some details: Remote system is not in my control. I only have SSH/rsync or FTP access Remote system…

问题原文

原帖链接 · jimmy0x52 · CC BY-SA 4.0

We have a large amount of files on a remote server that I'd like to setup regular backups to a local system for extra redundancy. Some details:

  • Remote system is not in my control. I only have SSH/rsync or FTP access
  • Remote system runs rsync 2.6.6 and cannot be upgraded
  • Remote system allows a max of 25 concurrent connections and 5 are reserved for production needs (so, 20 available)
  • Remote system contains 2M files - the majority of which are 100-200K in size
  • Files are stored in a hierarchy

Similar to:

0123456789/
        0123456
            abc/
                1.fff
                2.fff
                3.fff
            xyz/
                9.fff
                8.fff
                7.fff
9877656578/
        5674563
            abc/
                1.fff
                2.fff
                3.fff
            xyz/
                9.fff
                8.fff
                7.fff

with 10's of thousands of those root folders containing just a few of the internal folder/file structures - but all root folders are numeric (0-9) only.

I ran this with a straight rsync -aP the first time and it took 3196m20.040s. This is partially due to the fact that since the remote server is on rsync 2.6.6 I can't use the incremental file features found in 3.x.x. It takes almost 12 hours to compile the file list - running about 500 files per 10 seconds. I don't anticipate subsequent runs will take as long because the initial run had to download everything anew - however even 12 hours just for the file listing is too long.

The folder naming is broken up as such:

$ ls | grep "^[^67]" | wc -l
295
$ ls | grep "^6" | wc -l
14167
$ ls | grep "^7" | wc -l
14414

I've tested running this rsync -aWP --delete-during by breaking it up using --include="/0*/" --exclude="/*/" where I run 8 of these concurrently with 0* 1* 2* 3* 4* 5* 8* 9* and for 6 and 7 I use 60*-69* and 70*-79* because the brunt of the folders in the hierarchy begin with 6 or 7 (roughly 1400 per 6?* or 7?*).

Everything that's not a 6 or 7 takes about 5 minutes, total. The 6/7 directories (broken down in 1/10ths) take 15 minutes each.

This is quite performant, except to run this job I have to run 28 concurrent rsync and this saturates the available connection count - not to mention potentially saturating the network.

Does anyone have a recommendation for another variant of rsync or some additional options I could add to prevent this from using so many connections concurrently without having to stage this sequentially in the bounds of rsync 2.6.6 on one end?

Edit #1: We do pay for bandwidth to/from this external provider so ideally we would only send things over the wire that need to be sent, and nothing more.

已采纳解答

原帖链接 · jimmy0x52 · CC BY-SA 4.0

After an initial sync time of 40 hours to download and sync all of the data a subsequent scan and sync of the same data (just to pull in updates) only took 6.5 hours. The command used to run the rsync was:

rsync -a --quiet USER@REMOTE_SERVER:ROOT/FOLDER/PATH/ /LOCAL/DESTINATION

I think my large initial time for download was twofold:

  1. The initial dataset is 270GB and ~2M files, which is a lot to scan and download over the internet (in our case we have a 100mbit synchronous connection and this was connecting to a large CDN provider)

  2. I had the -P option enabled and -v options on the initial sync which caused a lot of local console chatter displaying every file being synced and progress information.

So, the answer here: Just use rsync with not so many verbosity options (and --quiet ideally) and it's quite efficient - even to huge datasets.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2019-01-22)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。