大网数据广告活动
服务器运维问答

Effectively handling 2+ million files

· 52CE 我爱测

I have a file based DB that has about 2M files stored in 3 levels of subdirectories. 2/2/6253 2/2/6252 ... File a vary from 30 bytes to 60 KB. Whole DB is Read Only. DB is about 125 Gigabytes big. Added: All files are compressed …

问题原文

原帖链接 · Worker · CC BY-SA 3.0

I have a file based DB that has about 2M files stored in 3 levels of subdirectories.

2/2/6253
2/2/6252
...

File a vary from 30 bytes to 60 KB. Whole DB is Read Only. DB is about 125 Gigabytes big.

Added: All files are compressed by zlib (python)

I want to handle it all as one file with file system in it. Which file system would be my best choose?

At the moment I use following script:

dd if=/dev/zero of=/my_file.iso bs=1024K count=60000
mkfs.ext4 -f /my_file.iso
mount -o loop /my_file.iso /mnt/

已采纳解答

原帖链接 · ewwhite · CC BY-SA 3.0

You probably just want to use XFS.

It's quite capable of what you're asking for, and does the job.

There's no reason to complicate this with lesser-used filesystems, which can come with other tradeoffs.

Please see: How does the number of subdirectories impact drive read / write performance on Linux? and The impact of a high directory-to-file ratio on XFS

If you want something more esoteric, ZFS zvols with a filesystem on top could provide an interesting alternative (for compression, integrity and portability purposes).

See here: Transparent compression filesystem in conjunction with ext4

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2015-05-04)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。