Tuning raidz2

[rsq]

Hi everyone,

I am looking for some tricks to make a raidz2 vdev as fast as possible on a ZFS on Linux. I am looking for single-stream sequential speed, random IO be damned.

Currently I have 8 WD red 3Tb drives in the system. When final, there will be 16 of these, in 2 Raidz2 vdevs. The vdevs will be in the same pool to leverage the striping. Pool will be created from scratch, so unbalanced pool will not be a problem.

The server is on a big fat UPS, so I am running with sync=disabled on the pool. I also have ashift=12 and atime=no set.

Write speed:

rsq@bitbucket2:/test$ dd if=/dev/zero of=/test/zeroes bs=128K count=819200
819200+0 records in
819200+0 records out
107374182400 bytes (107 GB) copied, 174.552 s, 615 MB/s

Read speed:

rsq@bitbucket2:/test$ dd if=/test/zeroes of=/dev/null bs=128K
819200+0 records in
819200+0 records out
107374182400 bytes (107 GB) copied, 288.926 s, 372 MB/s

Server specs:
Intel Xeon E3-1275-V2 4 Cores with HT @3500MHz
32Gb DDR3-1600 ECC mem
Ubuntu Server 14.04 LTS
ZFS installed from ppa:zfs-native/stable

I am open to all ideas for tuning. I am also wondering why my write speed is so much faster than my read speed.

Best regards, RsQ

 

[djflow195]

Build a raidz2 pool with 10 drives and try it again. You should have 2^n data disks not including parity.

Since you are just testing, just make a raid 0 pool and a raid 10 pool for comparison.

 

[paret0]

If you want the fastest Raidz2 pool possible on your hardware, don't run it on ZOL. It's just not as fast as native Solaris-based, or BSD implementations.

That being said, if your distro has DTrace (and I'm pretty sure it does), it may be worth a little midnight oil to optimize whatever you're running. Brendan Gregg is the DTrace Master http://dtrace.org/blogs/brendan

 

[danswartz]

All zfs implementations I know of optimizes writes of blocks of zeroes. e.g. it isn't really writing that much.

 

[SirMaster]

http://www.nerdblog.com/2013/10/zfs-xattr-tuning-on-linux.html

zfs set xattr=sa poolname

 

[HobartTas]

Try increasing the recordsize parameter to a larger size over its default 128KB, Solaris lets you go all the way up to 1MB.

Cheers

 

[danswartz]

Like I said, don't use zeroes.

 

[SirMaster]

Only the closed source Solaris version has Largeblock support.

They are currently working on Largeblock support for OpenZFS.

 

[kac77]

First thing would be to get the right number of disks for the raid level.
It should be 4, 6, or 10 for raidz2.

 

[danswartz]

No, for the 3rd time. First thing is to write a data pattern that will actually generate the expected disk traffic. Not zeroes.

 

[rsq]

Thank you for the input guys.

Currently running a comparative test with the xattr property of SirMaster. Will report back how it goes...

Danswartz, I could not find any source indicating that ZFS is optimized for writing zeroes. I know that compression=on makes a big difference with zeroes, but I have the compression disabled. Do you have a source for this?

It's not that I don't believe you, but I have been doing all my tests with /dev/zero, and it seems everyone else on the internet is also doing it. If my testing methodology is wrong, I would like to know it for sure.

I will make a small program that issues x number of 128K writes of random data. Using dd if=/dev/random seems to bottleneck on the CPU.

 

[madrebel]

use iometer or iozone if you want better source input. DD is ping for the IO subsystem. It is useful for proving disks are alive and useful to gather quick benchmarks to rule out hardware as the problem source.

beyond that though, not incredibly useful

 

[danswartz]

I meant to mention compression, but forgot until I had left for lunch, thanks for the reminder. I didn't see any reference to compression in your OP. Doing a couple of quick experiments, I see that the zero-write detection seems linked to compression. As to articles, I can't point to anything specific - just comments by various folks on mailing list. I think they may have referred to compression too. Even without compression, I'd guess the writes are faster since outbound data can be queued up in RAM on its way out. I saw much faster reads than writes on my raid10 (4x2) but then turned off all caching, and the read speeds went down to 1/3 of writes, which is more in line with what you saw. I see you have 32GB of RAM though - what is your ARC usage? If you re-read the file, does the next time go faster?

 

[rsq]

Read from ARC is hysterically fast, as expected:

rsq@bitbucket2:/test$ dd if=/dev/zero of=/test/zeroes3 bs=128K count=81920
81920+0 records in
81920+0 records out
10737418240 bytes (11 GB) copied, 15.7744 s, 681 MB/s
rsq@bitbucket2:/test$ dd if=/test/zeroes3 of=/dev/null bs=128K
81920+0 records in
81920+0 records out
10737418240 bytes (11 GB) copied, 1.13213 s, 9.5 GB/s

As is write of all zeroes with compression=on:

rsq@bitbucket2:/test$ sudo zfs set compression=on test
[sudo] password for rsq: 
rsq@bitbucket2:/test$ dd if=/dev/zero of=/test/zeroes4 bs=128K count=81920
81920+0 records in
81920+0 records out
10737418240 bytes (11 GB) copied, 1.9015 s, 5.6 GB/s

I have been testing with the xattr setting, but I get inconsistent results. I don't really trust the benchmarks. Looking into IOZone and IOMeter as suggested.

 

[rsq]

Using IOZone, I come to a more consistent result, for the write that is, rewrite is still inconsistent... WTF is going on here??

I used 320Gig (10x RAM) to rule out caching (not that caching should matter in writes) results are half of what I had using dd.

Test 1:

                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
       335544320     128 2960433 2224776

Test 2:

                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
       335544320     128 2995646 1505602

 

[kac77]

No, for the 3rd time. First thing is to write a data pattern that will actually generate the expected disk traffic. Not zeroes.

Was this addressed to my post? It would seem kind of silly to create a data pattern when the vdev itself doesn't have the optimum number of disks in it. You would be giving up performance for no apparent reason.

Also writing zeros isn't meant to find out optimum performance for a particular data set. It's meant to rule out issues that one could be having with hardware and to assess general performance.

For example, if all things are equal (tests are run identically) and you are getting 300 MB/s and generally you should get 600 MB/s then there's an issue somewhere. This would come long before configuring for a particular data set.

 

[danswartz]

No it wasn't aimed at you specifically. The point was if you have a data pattern which causes the disk write traffic to be greatly different than expected. the results are meaningless. In this case, a fixed (small) pattern with compression=on. No argument about the rest of your post. I think if one really wants an idea as to the performance characteristics of a pool, disabling compression (and probably also arc/l2arc caching) is a good idea...

 

[rsq]

I had compression disabled.

But even with compression disabled, the results of my benchmarks (iozone now) vary so much that I am unable to tell the impact of changing settings.

I've had runs of 700MB/sec with default settings, and runs of 450MB/sec with optimized settings. And vice versa.

 

[madrebel]

Using IOZone, I come to a more consistent result, for the write that is, rewrite is still inconsistent... WTF is going on here??

so re-write is a random pattern right. take blocks 4-10, read them, copy them, then write the new blocks. so whats occuring on the physical platters is you're taking blocks that were on sectors blah blah (towards the outer edge) and re-writing them to new sectors (will be on different/sectors tracks etc)

I used 320Gig (10x RAM) to rule out caching (not that caching should matter in writes) results are half of what I had using dd.

write caching is making a very big difference. idk if you're using iscsi or nfs for these tests but i guarantee sync is disabled and or writeback is enabled for the zvol. if you want to feel pain do zfs set sync=always pool/zvol or /pool/dataset. then if you're using a zvol you need to set ... actually in linux i'm not sure where you disable writeback.

anyway after forcing sync you'll feel the impact of no write cache (here im presuming you're not using a slog)

 

[devman]

I will make a small program that issues x number of 128K writes of random data. Using dd if=/dev/random seems to bottleneck on the CPU.

/dev/random is slow on Linux because it blocks to collect entropy in order to generate high quality random numbers for use in cryptography.

You want to use /dev/urandom which doesn't block and is more suitable for generating large volumes of psuedorandom data.