大网数据广告活动
服务器运维问答

Real-time file sync between servers with hunderd-thousands of small files

· 52CE 我爱测

I've given the task to create two CentOS 7 servers where not only the databases will be replicated but also files. Now my problem is that there will be probably hundred-thousands of files if not a million of files with a wide var…

问题原文

原帖链接 · Bert · CC BY-SA 3.0

I've given the task to create two CentOS 7 servers where not only the databases will be replicated but also files. Now my problem is that there will be probably hundred-thousands of files if not a million of files with a wide variety of sizes from a few Kbytes to ~1 Gbyte.

I've read about

  • incrion
  • lysncd
  • git-annex
  • ChironFS

Now I wish to ask your experiences about any of these if you have been using it or currently using it. How is the performance doing with the file changes regarding to copies and deletions? I'm very affraid of using any rsync because my experience is that it is not very fast with a lot of small files, therefore I can't really use it for a real-time file replication. Or am I wrong? Please prove me wrong. :)

Or maybe I'll need a 3rd and 4th server as fileservers? If yes, then the question still remains: How to replicate the files between the two servers in realtime?

Cheers!

已采纳解答

原帖链接 · shodanshok · CC BY-SA 3.0

If your servers are on the same LAN, then a clustered filesystem (ie: GlusterFS) or a shared storage solution (ie: via NFS) should be the better choice.

If your servers are in different location, having only WAN connectivity, the above solution will not work well. In this case, and if you only need one-way replication (ie: from active to backup server), lsyncd is a good solution. Another solution is csync2. Finally, another possibility is to use DRBD + DRBD Proxy (please note that its proxy component is a commercial plugin).

Finally, if your servers only have WAN connectivity and you need bidirectional replication (ie: both servers are active at the same time), basically no silver bullet exists. I'll list some possibilities, but I am far from recommending a similar setup:

  • unison with its real-time plugin
  • psync, which I exactly wrote for solving a similar problem (but please note that it has its own share of idiosyncrasies, and I provide no support for it)
  • syncthing with its real-time plugin (but it has significan limitations, namely it does not preserve ACLs nor the file's owner/group)

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2017-01-16)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。