raid-z performance better than zfs mirror?

M

__McB__

n00b
Joined
Apr 1, 2011
Messages
10
First off, I apologize if this has been discussed before, my search yielded nothing. Secondly, a lot of this is new to me so it is possible that I'm missing something rather glaring.

I am running SE11 with napp-it. When I run the dd bench via napp-it against my raid-z pool I see better performance than I do with a zfs mirror pool, using the same drives. Here are the results:

RAID-Z:
NAME STATE READ WRITE CKSUM
testpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
c9t0d0 ONLINE 0 0 0
c9t1d0 ONLINE 0 0 0
c9t2d0 ONLINE 0 0 0
c9t3d0 ONLINE 0 0 0
cache
c9t5d0 ONLINE 0 0 0
c9t6d0 ONLINE 0 0 0
write 10.24 GB via dd, please wait...
time dd if=/dev/zero of=/testpool/dd.tst bs=1024000 count=10000
10000+0 records in
10000+0 records out
real 29.7
user 0.0
sys 6.3
10.24 GB in 29.7s = 344.78 MB/s Write
________________________________________
read 10.24 GB via dd, please wait...
time dd if=/testpool/dd.tst of=/dev/null bs=1024000
10000+0 records in
10000+0 records out
real 30.8
user 0.0
sys 4.4
10.24 GB in 30.8s = 332.47 MB/s Read


ZFS MIRROR:
NAME STATE READ WRITE CKSUM
testpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c9t0d0 ONLINE 0 0 0
c9t1d0 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
c9t2d0 ONLINE 0 0 0
c9t3d0 ONLINE 0 0 0
cache
c9t5d0 ONLINE 0 0 0
c9t6d0 ONLINE 0 0 0

write 10.24 GB via dd, please wait...
time dd if=/dev/zero of=/testpool/dd.tst bs=1024000 count=10000
10000+0 records in
10000+0 records out
real 47.6
user 0.0
sys 5.8
10.24 GB in 47.6s = 215.13 MB/s Write
_______________________________________
read 10.24 GB via dd, please wait...
time dd if=/testpool/dd.tst of=/dev/null bs=1024000
10000+0 records in
10000+0 records out
real 32.6
user 0.0
sys 4.0
10.24 GB in 32.6s = 314.11 MB/s Read

As you can see, the write/read raid-z versus mirror difference is not huge (344/332 vs 215/314) but it does goes against everything I've read about mirror giving better performance.

Does this make any sense?
 
For the same # of disks mirroring will give you better random performance but not better STR that is at least in normal raid. I do not have a lot of experience with zfs.
 
The benefit of a mirror pool comes in the vdev multiplier for writes. A 10-drive pool built from mirrors should show noticeably better performance vs a 10-drive raidz/raidz2. With so few drives the impact is a bit less realized.
 
Again, I am not very familiar with all of this, but here are CDM results from my Win7 box to the SMB folder.

Raid-Z
raidz.jpg


Mirror
mirror.jpg
 
My theory: your raidz has 3 data drives, so large writes can be striped across 3 spindles, whereas your raid10 has only 2 spindles (logically.) I'd be curious to see what you get if your recreate the raidz with 3 drives total (therefore 2 data drives.)
 
I believe the random performance should be faster on the mirror. I am not sure why this is not happening.
 
danswartz said:
My theory: your raidz has 3 data drives, so large writes can be striped across 3 spindles, whereas your raid10 has only 2 spindles (logically.) I'd be curious to see what you get if your recreate the raidz with 3 drives total (therefore 2 data drives.)
Click to expand...

Give me 1 hour and I'll recreate it with 3 total drives and post the results.

FWIW, with 5 total drives, the raid-z results are even better:
423.14 MB/s Write
455.11 MB/s Read
 
Keep in mind that even doing random writes, if they are 512K, they will span multiple drives, so the same issue arises, IMO.
 
Here are the results from a 3 disk raid-z:

NAME STATE READ WRITE CKSUM
testpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
c9t0d0 ONLINE 0 0 0
c9t1d0 ONLINE 0 0 0
c9t2d0 ONLINE 0 0 0
cache
c9t5d0 ONLINE 0 0 0
c9t6d0 ONLINE 0 0 0

write 10.24 GB via dd, please wait...
time dd if=/dev/zero of=/testpool/dd.tst bs=1024000 count=10000
10000+0 records in
10000+0 records out
real 47.1
user 0.0
sys 6.0
10.24 GB in 47.1s = 217.41 MB/s Write
_______________________________________
read 10.24 GB via dd, please wait...
time dd if=/testpool/dd.tst of=/dev/null bs=1024000
10000+0 records in
10000+0 records out
real 43.4
user 0.0
sys 4.3
10.24 GB in 43.4s = 235.94 MB/s Read


The write speed is about the same as the mirror (217 v 215) but the read speed is less than the mirror (235 v 314).
 
from http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide

Should I Configure a RAIDZ, RAIDZ-2, RAIDZ-3, or a Mirrored Storage Pool?

A general consideration is whether your goal is to maximum disk space or maximum performance.
A RAIDZ configuration maximizes disk space and generally performs well when data is written and read in large chunks (128K or more).
A RAIDZ-2 configuration offers better data availability, and performs similarly to RAIDZ. RAIDZ-2 has significantly better mean time to data loss (MTTDL) than either RAIDZ or 2-way mirrors.
A RAIDZ-3 configuration maximizes disk space and offers excellent availability because it can withstand 3 disk failures.
A mirrored configuration consumes more disk space but generally performs better with small random reads.
If your I/Os are large, sequential, or write-mostly, then ZFS's I/O scheduler aggregates them in such a way that you'll get very efficient use of the disks regardless of the data replication model.
For better performance, a mirrored configuration is strongly favored over a RAIDZ configuration particularly for large, uncacheable, random read loads.

the conclusion:
for best sequential read/ writes:
raid-Z is a good choice

if you need IOP/s (VMware storage, databases) or best read/ write values with small files
ex for a webserver or a ESXi datastore: use as much mirrors as possible.
If you use it for ex.for a mediaserver or a backup-server, use Raid-Z.


Gea
 
This actually makes sense. Gea, I thought that large sequential reads won for mirrors too, since the reads can be spread between the spindles. His results seem to confirm that. I think my theory about the write speed has been confirmed too.
 
Thanks, again, Gea!

This will be used for storing VMs along with static archival data. So I guess my best option is to get a 6th drive and create 3 2-drive mirrors? Or is there a better way to set up the mirrors for maximum performance?
 
__McB__ said:
Thanks, again, Gea!

This will be used for storing VMs along with static archival data. So I guess my best option is to get a 6th drive and create 3 2-drive mirrors? Or is there a better way to set up the mirrors for maximum performance?
Click to expand...

for VMware, added mirrors are best.

On my main VMware SAN machines, i use 4 x 3 way mirrors build from Sandforce MLC SSD drives for best performance and
best security without cache drives (two failed are allowed - I have had some failures with my MLC drives.)

My read/write values are about 1 GB/s per dd and ESXi-VM's are really fast



Gea
 
Excuse my ignorance, but does a 3-way mirror give better performance than a 2-way mirror...or it is just more secure?
 
Keep in mind mirrors only help read performance. That said, a 3-way mirror is more reliable and the read speed may be higher than a 2-way too. But you are wasting 2/3 of your storage! Not saying there are no legitimate uses for it, but I would argue very few people need that... You would probably be best off creating a raid-z2 or something, IMO.
 
__McB__ said:
Excuse my ignorance, but does a 3-way mirror give better performance than a 2-way mirror...or it is just more secure?
Click to expand...

3 way mirrors give better read performance and security
(i need both for my main AD and web-server - disks are relatively cheap
and I do not trust MLC SSD unless i have used them for a longer time

currently i have about 10 failed drives of about 120 in 15 months)

Gea
 
Last edited:
You must log in or register to reply here.
Back
Top