RAID 6 using consumer/NAS drives?

Pultzar

Limp Gawd
Joined
Jun 22, 2010
Messages
198
I'm thinking about a 12x6 TB RAID 6 resulting in 60 TB total using HGST NAS drives. However due to the higher URE rating on these drives compared to enterprise drives,I'm concerned about failure during rebuild.

Has anybody tried such a configuration and done some rebuilds?

Thanks!
 
I would consider 12 drives being the absolute maximum for raid6 depending on how you are going to backup the array.
 
Not with the 6TB drives.
From my understanding, those drives are only rated for up to 8 drive configurations.
Do you have a special enclosure that they will be mounted in?

I have heard really good things about HGST's NAS drives. I have good experience with them as well, but to that extent.

I presume with a RAID that large, you will have a replication somewhere, correct? No way in the world would I leave a RAID that size without a backup, no matter which drives they are..

§
 
I wouldn't leave any storage config without a backup. ^ No matter what it is. :cool:
 
Personally with the availability of such large drives I've prety much retired raid 5/6 as an option and do only raid 10 now. Though, I want to also look at ZFS at some point.
 
I make three copies to LTO-6 and store them offsite.

But like others have said, I'm leaning RAID 10 for safety reasons. The data is video footage that is written one time and read many times. I'm also considering the new SMR drives.
 
I would consider 12 drives being the absolute maximum for raid6 depending on how you are going to backup the array.

Seems like it is about the RAID size rather than the number of drives? During a rebuild, all data must be read in order to reconstruct the data. Whereas with RAID 10, only data from one drive needs to be read which is a big win.
 
But like others have said, I'm leaning RAID 10 for safety reasons. The data is video footage that is written one time and read many times. I'm also considering the new SMR drives.

RAID-10 is not for reliability. It is for performance, mainly in random accesses. With RAID-6, any 2 drives can die without a failure in the array. In RAID-10, 1-(n/2) drives can fail without the array failing, depending on which drives. That means that sometimes, more drives can fail in RAID-10, but sometimes the second failure takes out the array. With RAID-6, you are guaranteed to withstand 2 failures.

If you are or can consider ZFS, another option is RAID-Z3, which is like RAID-6 except that it uses the equivalent of 3 drives, not 2, for parity (I say "the equivalent of" since the parity is distributed in RAID-5/6/Z*)

Otherwise.. maybe RAID-60? you can withstand 2 to 4 drive failures without the array failing. You'll have 4 disks at minimum eaten for parity, though.

You can also put a hot spare in there if you want. The advantage of a hot spare is that a good setup will immediately replace a bad drive with it. It won't help if a bunch of drives die at the same time, but since they rarely fail at the same time - there is usually at least a few hours or days between them - the hot spare can help get the array repaired ASAP before the next failure happens. If your setup does not automatically use the hot spare, it can still help you get the problem fixed sooner because you won't have to worry about getting a new drive physically installed to rebuild the array.
 
Last edited:
RAID-10 is not for reliability. It is for performance, mainly in random accesses. With RAID-6, any 2 drives can die without a failure in the array. In RAID-10, 1-(n/2) drives can fail without the array failing, depending on which drives. That means that sometimes, more drives can fail in RAID-10, but sometimes the second failure takes out the array. With RAID-6, you are guaranteed to withstand 2 failures.

If you are or can consider ZFS, another option is RAID-Z3, which is like RAID-6 except that it uses the equivalent of 3 drives, not 2, for parity (I say "the equivalent of" since the parity is distributed in RAID-5/6/Z*)

Otherwise.. maybe RAID-60? you can withstand 2 to 4 drive failures without the array failing. You'll have 4 disks at minimum eaten for parity, though.

You can also put a hot spare in there if you want. The advantage of a hot spare is that a good setup will immediately replace a bad drive with it. It won't help if a bunch of drives die at the same time, but since they rarely fail at the same time - there is usually at least a few hours or days between them - the hot spare can help get the array repaired ASAP before the next failure happens. If your setup does not automatically use the hot spare, it can still help you get the problem fixed sooner because you won't have to worry about getting a new drive physically installed to rebuild the array.

With RAID 10, the odds of a disk failure taking out the array is the same as RAID 1. It does not increase as the # of disks increases. With RAID 6 this is not true and that should be considered as well.

Not to mention how URE affects a RAID 6 rebuild.

ZFS is not really an option for me. I am on a Windows box and I need very high performance reads hence using a RAID controller directly.
 
"With RAID 10, the odds of a disk failure taking out the array is the same as RAID 1. It does not increase as the # of disks increases."

Say what? Of course the odds change (for the better.) Maybe you are thinking of raid 0+1?
 
Personally with the availability of such large drives I've prety much retired raid 5/6 as an option and do only raid 10 now. Though, I want to also look at ZFS at some point.

Nothing wrong with Raid6 for large arrays, as long as you have actual backups.

Raid 10 definetly is the way to go, but only one or 2 drive failure, either way on either array hope you have notification of drive failures.

Now, do you really need 60TB in 1 array?
 
"With RAID 10, the odds of a disk failure taking out the array is the same as RAID 1. It does not increase as the # of disks increases."

Say what? Of course the odds change (for the better.) Maybe you are thinking of raid 0+1?

I would agree with him... The chances of failure are related to how much redundancy there is and how it is arranged, and with RAID-10, it is always 50% and arranged in that same way, unless you want to get into using 3+ disks in each mirror.

I myself use RAID-10 (well, ZFS equivalent) which would certainly rebuild faster but I would rather have RAID-60 or something for this size. If you think RAID-6 is too slow then I think 60 is the solution rather than 10.
 
Last edited:
"With RAID 10, the odds of a disk failure taking out the array is the same as RAID 1. It does not increase as the # of disks increases."

Say what? Of course the odds change (for the better.) Maybe you are thinking of raid 0+1?

Raid 10 is the same with 4 drives or 40

2 drives failure on separate splits (the raid 0 or the raid 1)
if 2 drives fail on the same split = array is dead.
 
"With RAID 10, the odds of a disk failure taking out the array is the same as RAID 1. It does not increase as the # of disks increases."

Say what? Of course the odds change (for the better.) Maybe you are thinking of raid 0+1?

No I am talking about raid 1+0. As the number of drives increases, there are more chances that a drive will fail. However there are also more possibilities for the failure to be in a place where it does not take down the array. These probabilities cancel each other out.

Anyway my original question was regarding consumer drives and if people have done a rebuild at this size. Apparently not, which is what I was expecting.
 
If you're concerned about UREs, what are you doing to ensure data consistency at the moment ?

How do you know a file is good, how do you know a backup is good ?

With ZFS you can do RAIDZ3, a must with huge arrays in my opinion, since when one drive fails, you don't sweat as you can still withstand a failure without panicking. Also, a few errors on a drive will not be treated as a failure, only a really screwed up drive will be thrown off the array. LTO is good but you'd rather never have to restore, especially 60TB.

And you can do the equivalent of "RAID70", or even stripe 3 or more groups of RAIDZ3 vdevs, if the performance is needed..

I've nothing against RAID10 or the ZFS equivalent but its future seems to be in smaller arrays, maybe SSD arrays, where restoring would not be that problematic, and drive failures are less likely.
 
I'm not sure we're talking about the same thing at all. If you have a raid10 array with 10 drives, this is a stripe of 5 2-disk mirrors. If a drive has already failed, if a 2nd drive fails (assuming no commonality in cabling, hba, etc), the 2nd drive failure will only take out the array if it the other half of the already degraded mirror - e.g. there are 9 remaining healthy drives, only one of which is going to take out the array if it fails. e.g. approx 10% vs 50% for raid1. how is the the same?
 
I'm not sure we're talking about the same thing at all. If you have a raid10 array with 10 drives, this is a stripe of 5 2-disk mirrors. If a drive has already failed, if a 2nd drive fails (assuming no commonality in cabling, hba, etc), the 2nd drive failure will only take out the array if it the other half of the already degraded mirror - e.g. there are 9 remaining healthy drives, only one of which is going to take out the array if it fails. e.g. approx 10% vs 50% for raid1. how is the the same?

Having those extra drives has no effect on whether or not the drive that was mirrored with the first failure dies or not, but those extra drives do add their own chance of failures.

Like OP said, it actually does cancel out. He is correct about that.

With RAID-6, your chance of array failure DOES change if you add a drive because it's a fixed number of redundancy, rather than a percentage. With RAID-10, it is a percentage. That said - I keep pointing this out, not to sound like a broken record but just so the point doesn't get lost - I still wouldn't trust RAID-10 with a very large array. Unless, perhaps, the data on the array isn't all that important.
 
Last edited:
As I said above, I have three backups of my data.

Then raid6 would be fine. You would not care that much if you lost the entire array.

I would expect a software raid rebuild of around 1 day (linux software raid) with HW raid being at minimum 50% slower mostly due to the processors that exist in hardware raid controllers compared to a modern day cpu.
 
Last edited:
Yeah, I see your point. As far as array sizes go, I don't think anyone recommends raid10 for large numbers of drives. It's a huge win for random read-heavy workloads if you use ZFS, because you get much better IOPS.
 
Yeah, I see your point. As far as array sizes go, I don't think anyone recommends raid10 for large numbers of drives. It's a huge win for random read-heavy workloads if you use ZFS, because you get much better IOPS.

Over at Spiceworks it is the only thing that people seem to recommend for a large number of drives, since as you increase the number of drives you are not compromising safety.
 
Over at Spiceworks it is the only thing that people seem to recommend for a large number of drives, since as you increase the number of drives you are not compromising safety.

But you actually are, because it is still the case that your array can be toast on the second drive failure, whereas with RAID-6, 2 failures will be correctable. RAID-6 is more safe than RAID-10. It's RAID-5 that is severely overrated. RAID-6 is good stuff. I'd still prefer RAID-Z3 for a large array if I could.

RAID-10 can sometimes survive more failures than RAID-6, but you should always use worst-case-scenario figures, ESPECIALLY if each pair (mirror) of drives in your array is the same kind of drive from the same batch (meaning they are more likely to fail at the same time). For this reason I recommend that you use different drive models in pairs. Mine are (Seagate NAS + WD Red) striped with (Seagate NAS + Seagate Enterprise Value). 4 drive RAID-10-equivalent.
 
Then raid6 would be fine. You would not care that much if you lost the entire array.

I would expect a software raid rebuild of around 1 day (linux software raid) with HW raid being at minimum 50% slower mostly due to the processors that exist in hardware raid controllers compared to a modern day cpu.

Yeah I'm trying to be backup-first before worrying about the raid. It's not like the raid will protect me from theft or other damage to the system (including from software).

But I don't want to restore from a backup every 6 months if a drive fails as it can be a bit of a pain. Otherwise I would just go raid-0.

Speaking of backup I'm planning to add a HDD drive to the mix. So then I have multiple copies at multiple sites using two different technologies.
 
(I scrub weekly too.) My vsphere storage pool is 6 1TB nearline sas drives in raid10. I run veeam B&R doing nightly backups to the same pool (yes, I know, bear with me...) A different server has two 1TB drives in a raid1. Veeam B&R writes a duplicate of the backup to the 2nd pool on the 2nd server. Additionally, intra-day backup of the VMs is done by replicating the VMs to that backup storage pool (again, using veeam B&R). Monthly, I do a veeam backup to an external sas drive, as well as to amazon.
 
Last edited:
Back
Top