raidz = faster than mirror?

[TeeJayHoward]

I'm missing something here. Everything I've read tells me that a raidz pool's IOPS = vdevs*single-disk-IOPS. For mirrors, though, it's supposed to be disks*single-disk-IOPS. I just ran a test:

In one window, I started up the following command:

time dd bs=1048576 count=102400 if=/dev/zero of=/pool/100gb.file

In another window, I fired up a iostat, and watched.

zpool tank has 12 mirrors of 2 disks. 24 total disks.

zpool iostat -v tank 5|egrep -v 'mirror|c11'

                 capacity     operations    bandwidth
pool          alloc   free   read  write   read  write
------------  -----  -----  -----  -----  -----  -----
tank          50.4G  32.6T      0  7.51K      0   943M
------------  -----  -----  -----  -----  -----  -----

zpool pool has 45 disks in a single raidz3 configuration.

zpool iostat -v pool 5|egrep -v 'raidz3|c11'

                 capacity     operations    bandwidth
pool          alloc   free   read  write   read  write
------------  -----  -----  -----  -----  -----  -----
pool          10.9T  70.6T      0  10.2K      0  1.24G
------------  -----  -----  -----  -----  -----  -----

The only thing I'm really interested in here is writes, so that's all I tested. I expected to see the raidz3 cap out at ~100IOPS. Instead, I saw writes of over 10K. By contrast, I expected the mirrored pool to cap out at 2.4K IOPS. Instead, it hit 7.5K. So what's going on? Is my testing methodology flawed? Am I just not interpreting the results correctly? What?

 

[_Gea]

In your mirror example, you have 7,5k IOPS in 5s what means about 1,5k IOPS per s for 12 vdevs. Given that a single disk has about 100 IOPS: 100 x 12 = 1,2 K IOPS - good

In your Raid-Z3 example, you have 10,2k write operations in 5s = about 2k writes per s for one vdev where it should be only 100.

I suppose the reason is that your workload is more sequential not random where you can stream data (your heads need no repositioning for next writes) so the result is not so different from mirrors.

Over all the result is as expected.
For pure sequential writes performance scale with number of data disks. Only with small concurrent random read/writes performance scale with number of vdevs. In your case, you have a sequential workload with the effect that the Raid-Z is faster.

 

[patrickdk]

zfs was designed to do this on writes. to make all writes streaming.

Now attempt to re-read that data, expecially from a non-clean pool, and all your reads will be random.

You won't be able to get the 1000 iops per disk then, for streaming workloads, but the 100iops cause it will now be random.

 

[brutalizer]

Why do you have 45 disks in a single raidz3? Just curious, sounds like a very unorthodox configuration of your supermicro sc847 jbod case.

 

[paret0]

If you have DTrace Toolkit, you can run /usr/dtrace/DTT/iopattern to see individual disk I/O workload type