大网数据广告活动
服务器运维问答

Does KVM/QEMU support mdadm RAID5 properly?

· 52CE 我爱测

I'm not able to get anything close to usable performance using a guest file located on a mdadm RAID5 array. I've believed I optimized all the parameters of the array and filesystem for best R5 performance: set bitmap=none set str…

问题原文

原帖链接 · MrCalvin · CC BY-SA 4.0

I'm not able to get anything close to usable performance using a guest file located on a mdadm RAID5 array. I've believed I optimized all the parameters of the array and filesystem for best R5 performance:

  • set bitmap=none
  • set stripe-cache=32768 (tried from 256-32768)
  • EXT4 stride=128 / stripe-width=384 (chunk 512K, FS block 4K, 3 data-disk)

The array performance very good on the host (105MBs using no cache, 470MBs with cache). It's made of 4 x HDDs, relative slow.

  • It doesn't make any different if the image file is raw or qcow2
  • Tried both Virtio SCSI and Virtio SATA
  • Tried all cache combination, also in the guest itself (Windows 10 and Linux)

Does KVM/QEMU just not work very well with RAID5 mdadm arrays?

It's seem to be a latency problem (similar to what I've seen on ESXi with local drives).

Latency almost 17 seconds and on average write performance of 1-10MBs

An example from the virt XML:

<disk type='file' device='disk'>
   <driver name='qemu' type='raw' cache='writeback'/>
   <source file='/mnt/R5_DATA/data2.raw'/>
   <target dev='sdd' bus='scsi'/>
   <address type='drive' controller='0' bus='0' target='0' unit='3'/>
</disk>

<controller type='scsi' index='0' model='virtio-scsi'>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</controller>

Host: Debian 9 stretch (kernel 4.9.0-8-amd64)

已采纳解答

原帖链接 · MrCalvin · CC BY-SA 4.0

After doing a lot of testing it sure is possible to get very good performance. There are several circumstances that make a huge different:

  • Version of md (i.e. kernel), KVM/QEMU and virsh version
  • cache configuration for you guest (the cache setting in your virsh XML file)
  • MD configuration, especially the stripe_cache and chunk size (should be set low to 64K)
  • Optimize EXT4 filesystem, e.g. use noatime and depending on UPS backup etc. disable journaling and barriers.

I was running Debian stable (9, Stretch) and that mean you run a very old version of kernel, QEMU and virsh :-(

I now moved to Ubuntu stable for the same reason which will give you a huge jump to newer versions of those important components. In my view Debian stable it really a bad choice. When 10/Buster is released is guess it will be okay for a periode, but sooner or later you'll have an outdated system.

Secondly, it was very obvious that you need to use the page-cache of the host system. There are some advance caching going on when using MD RAID5. Even when the cache is full, the performance very good. In my case over 200MB/s write on slow disks which perform about 100MB/s in JBOD configuration.

Using cache=none in the XML and then use the page cache of Windows guest will give you very slow performance. SSD might be another story.

Remember also to play with the page-cache settings in Windows guest for the current disk to get the best result.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2019-03-10)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。