大网数据广告活动
服务器运维问答

Live file syncronization across multiple Linux servers with millions of files and directories

· 52CE 我爱测

What is the best way to synchronize huge data of a running production server? Our server has over 20 million files (small files with 10k and bigger files up to 50MB) stored in 1 millon directories. The size of all data is about 5…

问题原文

原帖链接 · Barmi · CC BY-SA 3.0

What is the best way to synchronize huge data of a running production server?

Our server has over 20 million files (small files with 10k and bigger files up to 50MB) stored in 1 millon directories. The size of all data is about 5 TB (steadily increasing).

Is it possible to synchronize the data with lsyncd and what are the limits (especially of inotify)? How much additional space does lsyncd need? What about the load (cpu and memory) and the live time?

Another solution would be GlusterFS. Is it possible to use GlusterFS on a production with none or minimal downtime? GlusterFS stores a lot of magic data in x-attributes files and the storage volume is about 15 to 20% bigger than systems with non GlusterFS. Seems like a huge amount of waste...? What about the load?

And at least rsync and cronjobs could do the job. rsync would only run on the slave... So no additional space is needed on the primary server, but rsync must read the full directory tree every time the cron runs...

社区高分解答

原帖链接 · ewwhite · CC BY-SA 3.0

I'd seriously recommend using something like ZFS for the filesystem.

Built-in tools like the ZFS snapshot and ZFS send/receive allow you to take block-level snaps of the filesystem and ship it to a second server.

Some third party tools like sanoid/syncoid can set automatic management/pruning and synchronization of your filesystem from one host to another.

This is done at the block device level, so you avoid the rsync checksum/inventory process.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2016-07-19)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。