问题原文
原帖链接 · Laurentiu Soica · CC BY-SA 4.0
I have a k8s cluster on 4 VMs. 1 master and 3 workers. On each of the workers, I use rook to deploy a ceph OSD. The OSDs are using the same disk as the VM Operating System.
The VM disks are remote (the underlaying infrastructure is again a Ceph cluster).
This is the VM disk performance (similar for all 3 of them):
$ dd if=/dev/zero of=testfile bs=1G count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 4.82804 s, 222 MB/s
And the latency (await) while idle is around 8ms.
If I mount an RBD volume inside a K8S POD, the performance is very poor:
$ dd if=/dev/zero of=testfile bs=1G count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 128.619 s, 8.3 MB/s
During high load (100% util for the rbd volume), the latency of the RBD volume is greater than 30 seconds.
I know that my setup is not what ceph recommends and that dd is not the best tool to profile disk performance, but the penalty from having ceph on top of VM disks is still huge.
VM Operating system is
CentOS 7.7.1908.
Kernel 3.10.0-1062.12.1.el7.x86_64
Network bandwidth between worker nodes:
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.00 sec 2.35 GBytes 2.02 Gbits/sec
Network latency is less than 1 ms.
I'm looking for some hints into further troubleshooting the issue and improving performance.
已采纳解答
原帖链接 · MaksaSila · CC BY-SA 4.0
It is not enough information about your CEPH cluster. But some thing will improve the performance:
- It is necessary to put the journal on separated SSD (NVMe is even better). Even if you SSDs.
- Use 10GbE network and separate cluster and external network. It will improve network latency.
- Don't use 3 copies volumes. It is nice feature, but it makes your cluster slower.
- By default, the scrubbing works all the time. It is necessary to change it. Better to do the scrubbing during the night.
- Use BlueStore as format for OSDs.
- Tune the server for maximum performance. For example, the CPUs governor should be performance.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
