RAIDZ2 disk configuration

[meatincereal]

In the ZFS Best Practices Guide there is an example of a RAIDZ2 configuration of 7+2 disks.

In the same guide, there is also a recommendation saying to use N(disks excluding parity) = 2, 4 or 6 and P(parity) = RAIDZ level (RAIDZ1: 1, RAIDZ2: 2, RAIDZ3: 3), where N+P = amount of disks. For RAIDZ2, 9 disks would not "fit" the bill.

How bad is it to use 9 disks? Do people follow this guide, or is it only there for guidance? What do I lose by going with a 9disk raidz2?

 

[blastingcap]

http://queue.acm.org/detail.cfm?id=1670144

 

[hotcrandel]

You get a performance hit, and there are already a number of factors that can affect ZFS performance pretty significantly, that you have to plan around.

I imagine it being something to the effect that the blocks of data cannot be "chunked" up properly as it is striped across the disks.

 

[_Gea]

In the ZFS Best Practices Guide there is an example of a RAIDZ2 configuration of 7+2 disks.

In the same guide, there is also a recommendation saying to use N(disks excluding parity) = 2, 4 or 6 and P(parity) = RAIDZ level (RAIDZ1: 1, RAIDZ2: 2, RAIDZ3: 3), where N+P = amount of disks. For RAIDZ2, 9 disks would not "fit" the bill.

How bad is it to use 9 disks? Do people follow this guide, or is it only there for guidance? What do I lose by going with a 9disk raidz2?

The background is simple.
If you have 2, 4, 8, 16,.. datadisks in a vdev, ZFS can stripe datablocks equally over all disks and your overall sequential performance is up to n x performance of a single disk.

If you have for example 9 disks (7 datadisks + 2 disks for Z2 redundancy) in a Raid-Z2, ZFS cannot stripe datablocks equally over all disks. This does not lower the performance below the value of the "good" Z2 vdev build from 6 disks (4 datadisks) but is a little lower than the math 7 x disk value.

In any case, additional disks give you more capacity and a higher sequential performance.
Such "unbalanced" situations are quite normal with ZFS. Each time when you add a new vdev or use vdevs of different size, you have this situation. I would not care about such "golden numbers" but more of efficieny of capacity vs costs vs i/o vs rebuild time vs total amount of slots.

For example.
My 16 slot backupservers are usually build from one Raid-Z3 vdev of 15 disks (capacity optimized)
My 16 slot fileservers are usually build from two Raid-Z2 vdev of 7 disks disks each + hotspare (Raid-60, more performance optimized)
My ESXi storage servers are either 4 x 3 way Mirrors (regular disks) or Raid 2xZ2 (Raid-60) build from SSD pools

I use max 15 disks to have one slot free for a disk replace.

On some servers I use 2 x Raid-Z2 vdevs, one from "newer" 2 TB disks and one from "older" 1 TB disks. In the near future i plan to replace the old 1 TB ones disk by disk with 2 or 3 TB disks to have more capacity without destrying the pool and maybee later the 2 TB disks with 4 or 8 TB ones. This is not "performance optimized" but very efficient in re-using disks or expanding pools on the fly without adding more disks.

 

[brutalizer]

I know lot of people that dont care about these recommendations. I mean, does it matter if I get 400MB/sec or 370MB/sec? It is plenty.

 

[Zarathustra[H]]

According to the performance testing in this (granted old) thread, it doesn't seem to matter.

Speeds seem to go up the more drives you add, irrespective of the n+partity recommendations.

I'm pretty new to this though, so I don't know for sure.

 

[danswartz]

Keep in mind for raidz*, to read a logical block, all drives must be read/written, so any speedup will tend to happen if you have sequential I/O (or multiple clients?)

 

[houkouonchi]

Keep in mind for raidz*, to read a logical block, all drives must be read/written, so any speedup will tend to happen if you have sequential I/O (or multiple clients?)

False (unless its degraded). Only small writes smaller than the stripe size on parity raid requires reading all drives before doing the write to re-calculate the parity on the block its writing to which is smaller than the striple. ZFS has ways of getting around this by not immediately writing it as parity raid and controllers with batteries can wait quite a bit and write things in a more sequential pattern when it does finally flush to disks avoiding this as well.

That being said the only time a read requires all drives to be read is when the array is degraded and thus has to reconstruct the data from parity (and that is the process that requires reading from all the disks).

Random reads on raid6 vs raid10/raid0 will be the same when the volume is in a normal state.

Sequential writes will be faster in raid6 than raid10 with more than 4 drives assuming you are not being bottlenecked by the CPU in the raid controllers or CPU on the machine (in the caise of zfs/mdadm).

Having an odd number of disks only matters in raid10 or raid10e type of setups and does not matter with parity raids like raid5/raid6/raidz/raidz2 because the parity is stripped between all the disks.

 

[Billy_nnn]

Remember though that raidz is not just raid5 by another name (and raidz2 is not just raid6 etc etc).

ZFS can behave quite differently from the more traditional raid types...

 

[danswartz]

False (unless its degraded). Only small writes smaller than the stripe size on parity raid requires reading all drives before doing the write to re-calculate the parity on the block its writing to which is smaller than the striple. ZFS has ways of getting around this by not immediately writing it as parity raid and controllers with batteries can wait quite a bit and write things in a more sequential pattern when it does finally flush to disks avoiding this as well.

That being said the only time a read requires all drives to be read is when the array is degraded and thus has to reconstruct the data from parity (and that is the process that requires reading from all the disks).

Random reads on raid6 vs raid10/raid0 will be the same when the volume is in a normal state.

Sequential writes will be faster in raid6 than raid10 with more than 4 drives assuming you are not being bottlenecked by the CPU in the raid controllers or CPU on the machine (in the caise of zfs/mdadm).

Having an odd number of disks only matters in raid10 or raid10e type of setups and does not matter with parity raids like raid5/raid6/raidz/raidz2 because the parity is stripped between all the disks.

Houko, this is kind of confusing, as most of your response seems to address standard raid, not raidz*. I dug up a post by richard elling who says that you may not need to read all N data drives to assemble the logical block, it depends on the recordsize (ZFS wise). If the recordsize is much larger than the disk sector size, you will end up reading all N data drives. Given the default zfs recordsize is 128KB, I'd say this is pretty likely.

 

[HobartTas]

Greetings

In the same guide, there is also a recommendation saying to use N(disks excluding parity) = 2, 4 or 6 and P(parity) = RAIDZ level (RAIDZ1: 1, RAIDZ2: 2, RAIDZ3: 3), where N+P = amount of disks. For RAIDZ2, 9 disks would not "fit" the bill.

How bad is it to use 9 disks? Do people follow this guide, or is it only there for guidance? What do I lose by going with a 9disk raidz2?

and

Houko, this is kind of confusing, as most of your response seems to address standard raid, not raidz*. I dug up a post by richard elling who says that you may not need to read all N data drives to assemble the logical block, it depends on the recordsize (ZFS wise). If the recordsize is much larger than the disk sector size, you will end up reading all N data drives. Given the default zfs recordsize is 128KB, I'd say this is pretty likely.

The answer is very simple, the recordsize is a power of 2 and the number of data drives should also be a power of 2, dividing one into the other leaves you with the data size to be written which is also a power of 2 and the end result is no wasted space, e.g. blocksize default 128KB / 8 data drives = 16KB written to each drive can be cleanly written with either 512 or 4K sectors, Sub-mesa previously discussed this topic but I can't find the original post, some discussion here

http://forums.freenas.org/archive/index.php/t-158.html

In the ZFS Best Practices Guide there is an example of a RAIDZ2 configuration of 7+2 disks.

NO! If you re-read that bit very carefully this example was for a very special case that overrides the general rule, It talks about a "For RAIDZ configuration on a Thumper" which AFAIK a Thumper was a Sun server with about 48 disks in it, apparently it was something like 6 hardware controllers with 8 drives on each one, hence it had 8 pools consisting of 5 data + 1 parity Raid-Z1 or 4 data + 2 parity Raid-Z2 or 3 data + 3 parity Raid-Z3. The reason that it had the non-standard pool sizes (for Z1 and Z3 cases but not Z2) was because if a controller died it took out 8 hard drives and this meant that it took out exactly one drive out of each pool so the 8 pools of 5 data +1 parity Raid-Z1 kept going (degraded but still functional). This was the sole and only reason for the deviation from the otherwise recommended number of ^2 data drives.

How bad is it to use 9 disks? Do people follow this guide, or is it only there for guidance? What do I lose by going with a 9disk raidz2?

Some wasted space because the non-standard data size to be written won't fit exactly into the 512/4K sectors and reputed performance degradation anywhere from as low as 5%-10% to as high as 50%.


Cheers

 

[danswartz]

HobartTas. this is a different issue. What I am talking about is whether when a client specifies a read, do all N data drives need to be read. As Richard points out, that depends on recordsize / sectorsize.

 

[houkouonchi]

Houko, this is kind of confusing, as most of your response seems to address standard raid, not raidz*. I dug up a post by richard elling who says that you may not need to read all N data drives to assemble the logical block, it depends on the recordsize (ZFS wise). If the recordsize is much larger than the disk sector size, you will end up reading all N data drives. Given the default zfs recordsize is 128KB, I'd say this is pretty likely.

ZFS Raidz and parity raids still have a lot of common. ZFS gets around the 'write hole' that normal hardware raid can not move around (to an extent).

The article you read if it was using words like 'assemble the logical block' that almost surely meant that they had to 'assemble' it because a device failed and thus had to rebuild/reassemble the data using the parity. If things are in a normal state it does *not* have to read from all disks for reads.

 

[danswartz]

That isn't how it works though. See this:

https://blogs.oracle.com/roch/entry/when_to_and_not_to

 

[houkouonchi]

That isn't how it works though. See this:

https://blogs.oracle.com/roch/entry/when_to_and_not_to

I don't see where they are talking about random reads only.

Anyway do you really think that ZFS is going to be worse than normal parity raid? Normal parity raid when doing random reads does not have to read from all disks at all. If it did results like this would be impossible in a 24 disk array:

Seeker v3.0, 2009-06-17, http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/sdd [85374958592 blocks, 43711978799104 bytes, 40709 GB, 41686991 MB, 43711 GiB, 43711978 MiB]
[512 logical sector size, 512 physical sector size]
[128 threads]
Wait 30 seconds..............................
Results: 2221 seeks/second, 0.450 ms random access time (823440211 < offsets < 43711355285469)

This is just random reads (512 byte seeks). I had some other disk activity going on or it would have been closer to 2500. That result is 92 random seeks/sec per disk which is very good. Raid6 also performs just as good as raid10 when it comes to random reads. The only times where non parity raids are faster is when its degraded (because then all disks have to be read) or when writing small random writes smaller than the stripe size.

 

[danswartz]

I wasn't talking about random reads, but about filesystem blocks being striped over all data members of a raidz* vdev (assuming the recordsize > sectorsize). You really need to stop confusing everyone with performance claims for standard raid and zfs, since they often don't match up. raidz2 is never going to be faster for random reads than raid10. If you're going to keep arguing with me about non-zfs raid, I'm done, since that was never what I was talking about, so it's confusing at best.

 

[danswartz]

You're also positing a configuration no zfs user would ever set up. e.g. a 24-disk zfs vdev. If someone was going to use that many disks in a raidz2, they would concatenate multiple raidz2 vdevs. Actually, though, if random reads were really critical, a raid10 config would be much better. I can't speak to standard raid levels - if you say raid6 can read from individual drives randomly, I won't argue with you, and yes, I'd guess you'd get great random IOPS, but raidz* does not have those characteristics. It *does* have load balancing across members of a mirror, which is why raid10 in zfs blows the doors off any raidz* for random reads.

 

[Billy_nnn]

I don't see where they are talking about random reads only.

Anyway do you really think that ZFS is going to be worse than normal parity raid? Normal parity raid when doing random reads does not have to read from all disks at all. If it did results like this would be impossible in a 24 disk array:
[

Raidz is not just raid5 by another name - you have to understand that first and foremost.
They are often compared, as the level of resilience to drive failure is similar - however, under the hood they work very differently - you simply can't assume that what may be true for raid5 is then also true for raidz - it often isn't. (ditto for raidz2 vs raid6 etc).

Without writing a whole diatribe, it uses copy-on-write rather than read-modify-write, variable blocksizes rather than fixed, and checksums every block read (traditional parity raid doesn't do any checking).

As has been pointed out, few zfs users would be likely to configure a pool with a single raidz/raidz2 vdev of 24x 2TB drives (and TBH I doubt I'd configure a traditional raid6 that way either, but each to their own ).
However, if they did, they'd likely find small random read performance would be some way off your figures (assuming this seeker utility is accurate of course - 512 byte reads from the raw device isn't really typical in the real world - it bypasses the filesystem layer and doesn't really reflect typical I/O patterns - it's a bit synthetic TBH) - in any case this type of I/O is not one of raidz's strengths.


PS - what is the small random write performance like?

 

[EnderW]

so what is the consensus on the optimal number of disks for a for RAIDZ2?
I was planning on 6x4TB for 16TB usable space before formatting

 

[danswartz]

6 disks.

 

[Aesma]

What would you do with 24 drives with max storage space as a main concern (with a full backup) ? I was thinking 2*12 drives RAIDZ2 or maybe RAIDZ3. For a home use with the most demanding task being streaming maybe two blu-rays, small file performance being of no interest. 4*6 RAIDZ2 is too much lost space for me.

 

[EnderW]

What would you do with 24 drives with max storage space as a main concern (with a full backup) ? I was thinking 2*12 drives RAIDZ2 or maybe RAIDZ3. For a home use with the most demanding task being streaming maybe two blu-rays, small file performance being of no interest. 4*6 RAIDZ2 is too much lost space for me.

well if

1) max storage space is the main concern
2) you have a backup
3) relatively low throughput performance needed

I'd think you could just do the ZFS equivelant of JBOD


my plan is to start with 6x4TB RAIDZ2 and add space later in groups of 6 disks
hopefully by the time I need more space, 5TB drives will be available

 

[houkouonchi]

You're also positing a configuration no zfs user would ever set up. e.g. a 24-disk zfs vdev. If someone was going to use that many disks in a raidz2, they would concatenate multiple raidz2 vdevs. Actually, though, if random reads were really critical, a raid10 config would be much better. I can't speak to standard raid levels - if you say raid6 can read from individual drives randomly, I won't argue with you, and yes, I'd guess you'd get great random IOPS, but raidz* does not have those characteristics. It *does* have load balancing across members of a mirror, which is why raid10 in zfs blows the doors off any raidz* for random reads.

I have seen lots of people who have made single vdev 24 disk raidz2 volumes so yes there *are* people who do that.

I didn't say raidzX or parity raid is faster than raid10 style raid for random reads. When the array/zpool status is normal I am saying that random reads are the same on both. I am sorry but saying all disks have to be read to read a block when the pool is in a normal state is simply INCORRECT. This would be true if the zpool was degraded.

I have heavily used ZFS and we have petabytes worth of storage on ZFS where I used to work. Its not like I know nothing about ZFS. I was just trying to point out something that someone said that was incorrect.

The article here:

https://blogs.oracle.com/roch/entry/when_to_and_not_to

Effectively, as a first approximation, an N-disk RAID-Z group will
behave as a single device in terms of delivered random input
IOPS. Thus a 10-disk group of devices each capable of 200-IOPS, will
globally act as a 200-IOPS capable RAID-Z group. This is the price to
pay to achieve proper data protection without the 2X block overhead
associated with mirroring.

With 2-way mirroring, each FS block output must be sent to 2 devices.
Half of the available IOPS are thus lost to mirroring. However, for
Inputs each side of a mirror can service read calls independently from
one another since each side holds the full information. Given a
proper software implementation that balances the inputs between sides
of a mirror, the FS blocks delivered by a mirrored group is actually
no less than what a simple non-protected RAID-0 stripe would give.

It is talking about random *writes*. Read iops will be just as good on raidz/raidz2 as a raid10 level raid (when the zpool is in a normal state).

 

[Aesma]

well if

1) max storage space is the main concern
2) you have a backup
3) relatively low throughput performance needed

I'd think you could just do the ZFS equivelant of JBOD

my plan is to start with 6x4TB RAIDZ2 and add space later in groups of 6 disks
hopefully by the time I need more space, 5TB drives will be available

I'm already doing a sort of JBOD but in Windows, with Liquesce (free pooling software). I want better safety than just a backup, maybe allowing me to move the backup offsite. With my current JBOD the drives are truly independent so if one fails it doesn't bring any other drive with it and I just have to get the corresponding backup drive, a pool without redundancy would crap entirely as soon as one drive craps, so it would be less safe than what I'm doing now.

 

[EnderW]

I'm already doing a sort of JBOD but in Windows, with Liquesce (free pooling software). I want better safety than just a backup, maybe allowing me to move the backup offsite. With my current JBOD the drives are truly independent so if one fails it doesn't bring any other drive with it and I just have to get the corresponding backup drive, a pool without redundancy would crap entirely as soon as one drive craps, so it would be less safe than what I'm doing now.

well if you have a backup, I wouldn't do any kind of large (6+ disk) array/pool with minimal redundancy because the risk of losing 2 disks and having to restore 24 drives worth of data is worse IMO than losing 2 disks and having to restore just the 2 drives of data that failed

RAID5/6 RAIDZ/2/3 are better suited for uptime IMO
situations where you have a lot of data that needs to be accessible at all times, even if a disk fails
not really applicable if you're just storing a bunch of movies for home use

we're getting a bit off topic here though and I would need more details on your setup to provide a better answer - if you make your own thread, PM me a link and I'll take a look

good luck

 

[Billy_nnn]

I have seen lots of people who have made single vdev 24 disk raidz2 volumes so yes there *are* people who do that.

They should be shot....

Seriously though, it's not that this won't work - of course it will - but there are practical considerations around performance, rebuild times, risk of double disk failure etc etc - this is fine if you understand/accept these implications, but this wouldn't be considered as a general use best practice.



As for the rest, as an example, in traditional raid, a 5 disk raid5 in non-degraded mode will behave similarly to a 4 disk raid0. Ditto for a 6 drive raid6.
This is not true with zfs - a 5 disk raidz (or 6 disk raidz2) will not behave the same as a 4 disk stripe. The former consists of a single multi disk vdev, whereas the latter is a stripe of 4 single disk vdevs - so the data is placed differently between the two.

 

[Swank]

I am sorry but saying all disks have to be read to read a block when the pool is in a normal state is simply INCORRECT.

...

The article here:

https://blogs.oracle.com/roch/entry/when_to_and_not_to



It is talking about random *writes*. Read iops will be just as good on raidz/raidz2 as a raid10 level raid (when the zpool is in a normal state).

Did you actually read the article you linked?

Now let's look at this from the performance angle in particular that
of delivered filesystem blocks per second (FSBPS). A N-way RAID-Z
group achieves it's protection by spreading a ZFS block onto the N
underlying devices. That means that a single ZFS block I/O must be
converted to N device I/Os. To be more precise, in order to acces an
ZFS block, we need N device I/Os for Output and (N - 1) device I/Os for
input as the parity data need not generally be read-in.

Back to basics
https://blogs.oracle.com/ahl/entry/what_is_raid_z

This actually means that RAID-Z is far more similar to RAID-3 where blocks are carved up and distributed among the disks; whereas RAID-5 puts a single block on a single disk, RAID-Z and RAID-3 must access all disks to read a single block thus reducing the effective IOPS.

 

[Billy_nnn]

Hmm, not sure about the raid3 analogy TBH - I think the guy is trying to make the point about raidz striping data across as many disks as possible, by varying the block size, but I think comparing to raid3 just confuses the issue, at least IMHO.

For raidz (N+X) variants, ZFS will try to stripe each filesystem block's data across as many vdev members (ie disks) as possible, up to N, but ZFS is a little smarter than traditional raid - in an N-way raidz it may decide to stripe individual filesystem blocks to fewer than N devices if it decides that fits better.

Generally though, it will stripe the fs block across the N devices, and must therefore read from N devices to get the block back - additionally for zfs, it must read the whole block in order to test that the block checksum is correct - a step which traditional raid doesn't do.