大网数据广告活动
服务器运维问答

Poor iSCSI performance with SSD disks and 10 Gbe network

· 52CE 我爱测

iSCSI Target Ubuntu 14.04 (Trusty Tahr) with 16 GB RAM and 16 core CPU as LVM backed iSCSI target using three Samsung SSD disks, each capable of doing 65k IOPS using an LSI 6 Gbit/s controller with on board cache. Benchmark on SS…

问题原文

原帖链接 · Kevin Parker · CC BY-SA 3.0

iSCSI Target

Ubuntu 14.04 (Trusty Tahr) with 16 GB RAM and 16 core CPU as LVM backed iSCSI target using three Samsung SSD disks, each capable of doing 65k IOPS using an LSI 6 Gbit/s controller with on board cache.

Benchmark on SSD disk in target:

fio --filename=/dev/sdd --direct=1 --sync=1 --rw=write --bs=4k --numjobs=10 --iodepth=1 --runtime=60 --time_based --group_reporting --name=ssd-max

iops=65514

Where sdd is configured in hardware RAID 0 using three Samsung 850 EVO SSDs.

Initiator

I exported a 500G LUN on an Ubuntu 14.04 client with 32 GB RAM and 8 core CPUs.

Benchmark on exported LUN

fio --filename=/dev/sdg --direct=1 --sync=1 --rw=write --bs=4k --numjobs=10 --iodepth=1 --runtime=60 --time_based --group_reporting --name=client-max

iops=2400

There is significant performance drop when doing DAS and over network, I was expecting at least 10k IOPS.

Communication between target and initiator is less than 1 ms and iperf shows a network throughput of 9.2 Gbit/s.

I understand that there will be a performance impact for 4k writes as each data has to go through the network stack of both the initiator and target before getting written to disk, but this is an unacceptable drop from 65k to 2k.

Where can the problem be? I have a 10 Gbit/s Ethernet NIC between the target and initiator. Any ideas?

已采纳解答

原帖链接 · shodanshok · CC BY-SA 4.0

Short answer: This is the results of network latency and a serial workload (as you imposed by using direct=1, sync=1 and iodepth=1).

Long answer: using direct=1, sync=1 and iodepth=1 you created a serial workload, as new writes can not be queued before the previous write was committed and confirmed. In other word, writes submission rate strictly depends on network latency. A simple ping between two machines can very well be in the excess of 0.2ms, more so when using a higher level protocol as TCP (and iSCSI on top of it). Presuming a total network latency of about 0.33ms, you have a maximum IOPS value of about 3000. This is without accounting for other latency sources (es: the disks themselves), so it is in-line with what you recorded.

Try this: execute a first benchmark without --direct=1 --sync=1, and another with these options in place but increasing the iodepth to 32 requests. Then report here the results.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2015-03-22)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0 / CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。