问题原文
原帖链接 · Ajay · CC BY-SA 3.0
So recently upon testing a ZoL system we found out poor performance of random and sequential reads and poor performance of random writes on our SSDs.
Our system is a stripe of 2x Samsung 1TB 850Evo SSDs to test ZFS performance and it was abysmal compared to LVM: reads are slower than HDDs and writes are not on par of expected 1.7GBs we get on LVMs. It's weird because our FreeBSD backup server has slow HDDs and older type of SSD and performs better at the same test.
The system is somewhat deprived of RAM though (zfs gets 4gb for arc and everything else is taken by VMs) however with no cache and no sync the performance is still not even close to anything.
So we are looking into buying newer systems based on AMD Epyc and setting up either full NVMe or NVMe with SSDs with disabling cache to free up ram from ZFS at least a bit (we want it to use 10GB max for everything). We don't really need all of security features of ZFS apart checksumming (but with SSDs it seems to be redundant as SSD run an inside checksumming system) so SSDs will be a stripe of vdevs.
We prefer ZFS for zle on thin-provisioned zvols and the ease of snapshotting and incremental backups to remote system (which runs ZFS as well).
However the struggle for performance is hard...
Would greatly appreciate any advice
已采纳解答
原帖链接 · Ajay · CC BY-SA 3.0
If someone wonders. We think the main problem is RAM (our ARC is limited at 4GB so and everything else is eaten by the system). The deal with ZFS at the moment - it is not ready for SSDs and/or NVMe. It has been made for HDDs, slow and bulky with their silly heads, mechanics and predictable issues.
With SSDs and NVMe ZFS performs silly things they don't need and does not do things they actually need. Back when ZFS has been invented nothing more of cache has been thought of non-volatile RAM.
Now we can put 4x pcie SSDs in a system with 4TB of space.
There are 2 ways to handle this in such case. Either give it enough memory and let it perform properly on your SSDs with the overheads it provides. Or not use ZFS.
It's a shame because its structural benefits are pretty good. But it can't handle SSDs properly without higher RAM usage than with HDDs because all the settings and configuration tell it "underlying systems are slow, cache is necessary, read small, write big and sequential" when SSDs are fast, don't need cache, can read big and write big and do random properly. With Optane such problems will be obvious.
Things that are more-or-less not needed are extensive caching, checksumming of each file at the record level (it doesn't make sense because if you have bitrot at SSD level you should throw away the whole drive as there are no uses for such system because it can have broken controller ruining your whole data, it's similar to bad RAM). SIL is not needed at all. ARC is also useless, especially with Optane drives (it adds more overhead to CPU and RAM). Record size should be perfectly limited to write in transactions drive understands.
Or just use LVM for KVM provisioning on systems. Thin-provisioning is not perfect there but at least you don't need to waste extremely valuable RAM on making your SSDs perform at the level they are supposed to.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
