Confused about ZFS performance on SSD

bleomycin

Limp Gawd
Joined
Aug 14, 2010
Messages
242
tldr: What is going on with these benchmarks, and do I really need a P3700 to get decent performance for VM's hosted on SSD ZFS pools? Ceph: how to test if your SSD is suitable as a journal device? | Sébastien Han

So I have a proxmox host that currently has the following configuration:

2x240GB mirrored ZFS Sandisk Extreme II SSD's
2x480GB mirrored ZFS Sandisk Extreme II SSD's

The 240GB drives are mostly empty, hosting just proxmox itself, and my VM's live on the 480GB mirror. I recently noticed that samba file transfers under one of my Windows 10 VM's were stalling when copying files from my NAS to itself over 10Gbit.

I ran this simple DD write test (with compression disabled on the filesystem) on the 480GB pool (which I know isn't terribly accurate):

Code:
dd if=/dev/zero of=tempfile bs=1M count=4024 conv=fdatasync,notrunc
4024+0 records in
4024+0 records out
4219469824 bytes (4.2 GB) copied, 35.0058 s, 121 MB/s

I saw as low as 80MB/s in further tests. Chalking this up to the lack of trim in ZoL and the fact the pool became probably close to 80% full at one point I destroyed the 480GB pool and forced a trim on them with:

Code:
mkfs.ext4 -F -E discard /dev/sda

And performance restored to ~450+MB/s or so in the same tests when I rebuilt the mirror. I wasn't able to secure erase these drives because they are frozen and in a machine I don't have easy physical access to, so trim was the best I could do.

Long story short i'm considering getting some newer SSD's that will perform better (especially without TRIM) and stumbled across this post showing benchmarks for a ton of drives:
Ceph: how to test if your SSD is suitable as a journal device? | Sébastien Han

I'm horribly confused about what's going on there. How is an 850 Pro only writing at 1.5MB/s and how will that affect someone like me, a hobbyist without very demanding needs? I just want a system that's reasonably quick and responsive, but by looking at that page you'd think you need nothing less than a P3700 to pull that off?

Thank you for any help!
 
I actually think that the low write rate of the Samsung consumer SSDs is a rather good sign for the firmware implementation of the drives. When a cheap consumer class SSD like the Kingston v300 or the Intel 535 can achieve such a high sustained random 4K performance I get skeptical. Such a throughput without a proper power-loss-protection most probably means that the SSD does a heavy write-caching in its RAM, which would mean that a power loss can have bad implications on your file-system integrity - something ZFS cannot deal with very well.

I run some VMs on an mirror of Samsung 830 256 GB SSDs for several years now and the performance is still good. While you can really see the performance degrading a bit during the time the SSD fills up and internally fragments, you can counter that a bit with overprovisioning (I use only 200 GB). The 830 was an early SSD and the garbage collection seems to be not as good as the 840 Pro I also testet. It is still enough for my home server.

Whether this is enough for you depends on your requirements - number of VMs, expected workloads, etc.
 
Back
Top