ZFS: RAID-Z3 (raidz3) Recommended Drive Configuration

Would raidz3 have performance advantage over raidz2 (if I ran raidz3 with 4 drives vs raidz2 with 3 drives)?

P1x3L said:

Would raidz3 have performance advantage over raidz2 (if I ran raidz3 with 4 drives vs raidz2 with 3 drives)?

Click to expand...

VDEV drive count is more important to sequential read speed than the redundancy level.

Actually, given the same drive count, having more redundancy should slow things down, not speed it up. You may not notice it much on a system with ample CPU power, but on a system with a slower CPU, you'll spend more time calculating the checksums. Roughly how it works is, on a RAIDz VDEV each block read/written has one checksum. On a RAIDz2 VDEV each block has two checksums, etc. etc. So, RAIDz3 will have 50% higher CPU load than RAIDz2.

For performance what you really want to do is split your data between multiple VDEV's in the same pool.

My main storage server at home which is constantly being hammered with time critical requests has dual 6 drive RAIDz2 VDEV's. In this case ZFS stripes together the two VDEV's into the same pool making it more parallelized.

On my remote backup server which is only used to receive backups (unless something goes wrong) is limited by network speed for its only use. For it, since I thus didn't care much about performance, I just mashed all the drives together into a single large 12 drive RAIDz3 pool to maximize storage efficiency and redundancy.

Zarathustra[H] said:

My main storage server at home which is constantly being hammered with time critical requests has dual 6 drive RAIDz2 VDEV's.

Click to expand...

So that is 12 drives across 2 vdevs? And each vdev is in raidz2? And 2 vdevs in 1 pool? Just trying to make sure I understand.
Thanks.

P1x3L said:

So that is 12 drives across 2 vdevs? And each vdev is in raidz2? And 2 vdevs in 1 pool? Just trying to make sure I understand.
Thanks.

Click to expand...

Yeah,

That's how I do it. But there are many ways to skin a cat. I tend to like this one.

A bit off topic, but I'm getting good info here... Is it possible to hotswap/connect/disconnect SATA drives if the ZFS pool they're on is unmounted first? No need to turn off the computer, right? Is there an accepted process for this?

P1x3L said:

A bit off topic, but I'm getting good info here... Is it possible to hotswap/connect/disconnect SATA drives if the ZFS pool they're on is unmounted first? No need to turn off the computer, right? Is there an accepted process for this?

Click to expand...

No need to unmount the pool.

It depends on your storage controller/motherboard though. It has to support hot swapping.

You can replace drives while the pool is active up to the redundancy level. With RAIDz3 you can remove up to three drives while the pool is running and doing its thing. Replace the drives and resilver it. It will slow down a little during the resilver, but other than that, no problem.

It's generally not advisable to remove drives to the point where you have no redundancy left though. So with RAIDz3, it's probably best to only do two drives at a time. With RAIDz2, one drive at a time. I wouldn't run Raidz. Single drive redundancy is for all intents and purposes obsolete at this point.

Zarathustra[H] said:

No need to unmount the pool.

It depends on your storage controller/motherboard though. It has to support hot swapping.

You can replace drives while the pool is active up to the redundancy level. With RAIDz3 you can remove up to three drives while the pool is running and doing its thing. Replace the drives and resilver it. It will slow down a little during the resilver, but other than that, no problem.

It's generally not advisable to remove drives to the point where you have no redundancy left though. So with RAIDz3, it's probably best to only do two drives at a time. With RAIDz2, one drive at a time. I wouldn't run Raidz. Single drive redundancy is for all intents and purposes obsolete at this point.

Click to expand...

I hot swapped a drive a few weeks ago. As the man says, it depends on your gear. I also agree with not removing ALL parity drives at once. I disagree about his assessment of Raidz. I would use it with a solid state NAS or SAN, and be comfortable doing so - but if resources allow, more is always better. It just has a cost associated with it.

I would also be comfortable using just one parity drive in a backup server as well...probably.

I should have been clearer. What about having a 5-drive pool that is all external SATA drives that you don't want to have running all of the time. So without turning off your computer, you can safely detach their power & SATA after unmounting the pool they're a part of?

That works
- as long as your Sata chipsets supports hotplug and it is enabled in the bios. Sometimes you also need to enable Sata hotplug in your OS of choice.

john4200 said:

RAID-Z3 is better than RAID 10 in most cases that do not require high performance for small, random writes. Certainly for a media server, the performance of RAID-10 is not required.

First, with 10 drives of capacity C, you get only 5C available capacity with RAID 10. You get 7C with RAID-Z3. That is 40% more space.

Then there is the probability of data loss.

...

So, for a capacity of 8C, you need only 11 drives with RAID-Z3, as compared to 16 drives with RAID-10, and your probability of data loss during a rebuild after a single drive failure is only 1% with RAID-Z3, as compared to 5% with RAID-10.

If F were only 2%, then the corresponding data loss probabilities for RAID-10, -Z1, -Z2, -Z3 are: 2%, 14.9%, 1.3%, and 0.086%. So that is less than 0.09% for RAID-Z3, as compared to 2% for RAID-10 : 23 times higher chance of data loss with RAID-10 as compared to RAID-Z3 !

Bottom line is that RAID-Z3 is greatly superior to RAID-10, except in rare applications where the higher performance of RAID-10 is required.

Click to expand...

Old post but since the thread already got necroed..
You didn't take into account that the rebuild process on the RAID10 takes a fraction of a time compared to Z3, and, as you've said, that is when the drives are hit the hardest and have the highest chance of failure. I would take mirrored stripes with a hot spare over Z3 any time the loss of capacity isn't a deal breaker because of random I/O performance benefits AND a much easier capacity expansion. I can always buy a pair of drives at a time, but 7-11 at once is cost prohibitive.

I would agree. I usually prefer RAID10 with proper backups over Z3. Performance wise it is much more important for me.

Schmeisser said:

I would agree. I usually prefer RAID10 with proper backups over Z3. Performance wise it is much more important for me.

Click to expand...

I use dual RAIDz2 at home for my near line performance.

For my colocated remote backup server that only ever sees WAN speeds, I figured, what the hell, and went with the better storage density of RAIDz3.

I haven't found the rebuild times to be crazy, but I do have a pretty overkill (for the purpose of a backup server) dual socket Xeon with a total of 8C/16T and 96GB of RAM, so maybe that is why.

I just moved from 2x 6-disk Z2 to 1x 12-disk Z3. I changed the record size to 1024K.

Overall I gained 1 disk of usable space back.