大网数据广告活动
服务器运维问答

Clever way to reduce rsync/rclone load with backups to Amazon S3?

· 52CE 我爱测

I started running rclone to keep a set of files on my laptop that are synced up to the cloud and encrypted. I wanted to 'control' the encryption because of trust issues. Rclone does this pretty well, and so well that I use it for…

问题原文

原帖链接 · Not a chance · CC BY-SA 4.0

I started running rclone to keep a set of files on my laptop that are synced up to the cloud and encrypted. I wanted to 'control' the encryption because of trust issues. Rclone does this pretty well, and so well that I use it for a lot of my stuff, but that means that the number of files in there is getting bigger and that is costing me money.

I thought about alternatives to this, like syncing just the difference in a zip or an rdiff file or something. It is clear that this could work but it has some disadvantages or hurdles to manage.

First of all, the S3 storage isn't a real rsync server in the sense that it does not only copy the part of the file that changed. If I want to change to a system that sends a blob up to the cloud it needs the intelligence to know what is already up there.

The features I am looking for are:

  • Minimal connection load for backups,
  • Ability to quickly restore when I stupidly delete a file.
  • Low number of files stored on server
  • Full or near full file history
  • Something I can make a script for that takes the load off my brain to figure out all the options I need to include when syncing.
  • Ideally something that works with multiple devices although I don't use multiple devices simultaneously.
  • I should control the encryption (done locally) and keys

I'm posting this here because I thought this question would produce a worthwhile answer for a lot of people with server-type usage requirements.

Thanks in advance

[EDIT: Added that I want full control of encryption keys and encryption should be done locally before transfer]

已采纳解答

原帖链接 · rvs · CC BY-SA 4.0

I'd recommend to look for backup software that supports S3, and almost all of them are.

Two following I've used in the past or using now - with Google's GCS but they also support S3:

  • Duplicity allows incremental backups, although requires full backup once in a while
  • Duplicacy is a fancy once and allows deduplication across machines. Although it seemed a bit less stable for me, i.e. sometimes my snapshots will be missing a few files.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2018-12-25)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。