大网数据广告活动
服务器运维问答

Best block size for fio when reading a file

· 52CE 我爱测

To pre-warm an ext4 EBS volume I'm using fio as follows: fio --name <filename> --filename <filename> --rw read --direct 1 --ioengine libaio --bs <X>k and I'm trying to understand what the optimal block size should be. I know that…

问题原文

原帖链接 · XCore · CC BY-SA 4.0

To pre-warm an ext4 EBS volume I'm using fio as follows:

fio --name <filename> --filename <filename> --rw read --direct 1 --ioengine libaio --bs <X>k

and I'm trying to understand what the optimal block size should be. I know that I can 'stat' and get the block size of a file but when using that in fio, it will throw an error if the file size is less than its block size.

An option could be to use the block size given by stat by default and if the file size is less than that, get the closest 'standard' number: i.e. if size is less than 4k set block size to 1024.

What's the best way of setting the right block size?

EDIT: I'm restoring a 10TB gp2 volume from a snapshot. There are a few million files - most of them are small files, but another good part is made of 50MB-30GB files and all these files need to be "ready" to be read as fast as possible. I've got a script that runs fio against each file and I'm trying to understand how I can best dynamically adjust the block size for each.

已采纳解答

原帖链接 · shodanshok · CC BY-SA 4.0

In order to mask network latency, you want to use a reasonably large block size. The Amazon-suggested 1 MB block size seems good to me.

I suspect dd would be as fast, or faster, than fio for this particular workload. However, you simply had to experiment and use whatever method is faster for reading (and re-hydrating) the volume.

Finally, consider that stat returns two I/O size values:

  • minimum, which is the minimum IO size the device will read/write;
  • optimal, which is the minimum IO size to get good performance by avoiding r/m/w behavior

This does not means that IO bigger than optimal size will be slower; rather, bigger size can actually slightly increase IO performance.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2019-09-24)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。