ZFS Pool with multiple RAIDZ - Have I got this right?!

D

DigitalDJ

n00b
Joined
Sep 11, 2010
Messages
9
OK,

Quick but simple query (I think). From the reading I've done on ZFS...the picture in my mind tells me that I could have a single zpool and in that have multiple sized RAIDZ2.

For example, if I had 10x 2TB, 6x 1.5TB and 4x 1TB, I could have a single zpool that contains a 3 RAIDZ2s of 10x 2TB, 6x1.5TB and 4x 1TB respectively...

As my first dive into ZFS I tried to test this out with the following commands:

1. zpool create nas raidz2 <drive1-10 2TB>
2. zpool add nas raidz2 <drive1-6 1.5TB>
3. zpool add nas raidz2 <drive1-4 1TB>

As soon as I try the second command ZFS tells me:

invalid vdev specification
use '-f' to override the following errors:
mismatched replication level: pool uses mirror and new vdev is raidz

Now...should I be doing this? Or is this just a warning? When I force it the zpool creates successfully and I have different RAIDZ2s under the single zpool.

By doing this, I am losing out on any space? (Other than obviously having 6 different parity drives) i.e. in this example should I be getting the maximum benefit of 8x 2TB, 4x 1.5TB and 2x 1TB?

Also, I haven't tried this out with my actual drives yet...currently I've just used my 10x 2TB drives to experiment (a 5x 2TB RAIDZ2 and 5x 2TB RAIDZ2). Already, I can see performance drop when I run bonnie++ by quite a bit. Is this expected? Or am I setting it up wrong?

Cheers,
Grant
 
You might not want to do a 8 drive RAID-Z2; maybe do 2 4-drive RAID-Zs; same usable capacity, better IOPS, less chance of death in a resilver/rebuild.

That syntax I believe is correct; essentially you want one zpool tank, made up of various vdevs (in this case all your various drives). What will really help out with a myriad of differently sized vdevs is a SSD that will serve as ZIL & L2ARC, from what I've read/been told.
 
DigitalDJ said:
OK,

Quick but simple query (I think). From the reading I've done on ZFS...the picture in my mind tells me that I could have a single zpool and in that have multiple sized RAIDZ2.

For example, if I had 10x 2TB, 6x 1.5TB and 4x 1TB, I could have a single zpool that contains a 3 RAIDZ2s of 10x 2TB, 6x1.5TB and 4x 1TB respectively...

As my first dive into ZFS I tried to test this out with the following commands:

1. zpool create nas raidz2 <drive1-10 2TB>
2. zpool add nas raidz2 <drive1-6 1.5TB>
3. zpool add nas raidz2 <drive1-4 1TB>

As soon as I try the second command ZFS tells me:

invalid vdev specification
use '-f' to override the following errors:
mismatched replication level: pool uses mirror and new vdev is raidz

Now...should I be doing this? Or is this just a warning? When I force it the zpool creates successfully and I have different RAIDZ2s under the single zpool.

By doing this, I am losing out on any space? (Other than obviously having 6 different parity drives) i.e. in this example should I be getting the maximum benefit of 8x 2TB, 4x 1.5TB and 2x 1TB?

Also, I haven't tried this out with my actual drives yet...currently I've just used my 10x 2TB drives to experiment (a 5x 2TB RAIDZ2 and 5x 2TB RAIDZ2). Already, I can see performance drop when I run bonnie++ by quite a bit. Is this expected? Or am I setting it up wrong?

Cheers,
Grant
Click to expand...

You can have one pool which contains 3 vdevs of various drive sizes and replication levels.
If your new to this may I suggest reading this http://hardforum.com/showthread.php?t=1521803&page=17 thread and downloading Mesa's 1.7 preview system. You should also read http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide where you will learn the theory behind proper ZFS setup.

Also: http://constantin.glez.de/blog/2010/02/seven-useful-opensolaris-zfs-home-server-tips
http://constantin.glez.de/blog/2010/06/closer-look-zfs-vdevs-and-performance
http://constantin.glez.de/blog/2010...ove-oracle-solaris-zfs-filesystem-performance
 
Why would you want a RaidZ2 with only 4 drives? (your 3rd set)

You are going to lose 2 drives to RaidZ2, so might as well make it a Mirror set... less parity calculations etc prob faster as well.

I would go

5x 2TB RaidZ
5x 2TB RaidZ
6x 1.5TB RaidZ
4x 1TB RaidZ

Don't forget ZFS Stripes across these sets for speed,:)
 
movax said:
You might not want to do a 8 drive RAID-Z2; maybe do 2 4-drive RAID-Zs; same usable capacity, better IOPS, less chance of death in a resilver/rebuild.
Click to expand...

The Raid Z2 will actually have a much, much longer mean time to data loss than the striped pair of Raid Z vdevs. Double parity adds much more data protection than just using two smaller single parity arrays.
 
wingfat said:
You can have one pool which contains 3 vdevs of various drive sizes and replication levels.
If your new to this may I suggest reading this http://hardforum.com/showthread.php?t=1521803&page=17 thread and downloading Mesa's 1.7 preview system. You should also read http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide where you will learn the theory behind proper ZFS setup.

Also: http://constantin.glez.de/blog/2010/02/seven-useful-opensolaris-zfs-home-server-tips
http://constantin.glez.de/blog/2010/06/closer-look-zfs-vdevs-and-performance
http://constantin.glez.de/blog/2010...ove-oracle-solaris-zfs-filesystem-performance
Click to expand...

Hi,

I found the ZFS vdev link most useful.

I haven't had much trouble with all this stuff...it all makes sense..The only problem is that I've seen tons of tutorials that add multiple vdevs to a single pool but not one mention the error message that I received. In fact, that link mentions that adding multiple vdevs is fine and striping occurs between them.

A little Googling shows an Oracle page that specifically does not recommend having multiple vdevs of a different number of drives...

mismatched replication level: pool uses 4-way raidz and new vdev uses 5-way raidz
http://docs.sun.com/app/docs/doc/819-5461/gazgc?l=en&a=view

Which has me a bit baffled...The Oracle page doesn't really specify why but many other of the links say it's fine....although I've never once seen the error message mentioned. So is it safe to ignore and just use -f?
 
Steve01S4 said:
The Raid Z2 will actually have a much, much longer mean time to data loss than the striped pair of Raid Z vdevs. Double parity adds much more data protection than just using two smaller single parity arrays.
Click to expand...
Indeed. Imagine the possibility of losing 2 disks. With 2 raidz vdevs, you have a 50% chance of losing all your data, since both failed drives can be on the same vdev. Raidz2 protects you from that. The fact that raidz2 takes much longer to resilver adds a small chance of both disks failing at the same time but its reliability is still orders of magnitude better than raidz.

Good read:
http://blog.kj.stillabower.net/?p=93

To sum up: if you care about your data, always use multiple raidz2 vdevs of no more than 10 drives each. If you have a quad xeon and dedicated bandwidth for each disk (no SAS expander bullshit) you can use more drives for each vdev.
 
You must log in or register to reply here.
Back
Top