Optimum number of drives before using RAID6

Colonel_Panic

Limp Gawd
Joined
Oct 5, 2009
Messages
328
I'm evaluating my new storage options while my current server fills up, and have to make a decision. I'd like to build a system with about 9 drives in the pool but I can't decide what level of hardware RAID I should use. Since I'll have more than the necessary numbers for RAID 5 (3) and RAID 6 (4), I have to decide if losing a second drive to parity would be advantageous.

Statistically, with what number of drives does RAID 6 become beneficial? Additionally, is there a point where a failure is all but inevitable?
 
May I pose a question to you ; Why are you using RAID? I think its beneficial for us to know your situation first. Storing video/media? Storing your clients databases? Virtual Machines? Development?

I am not "Anti-RAID" but I think its can be misused in situations.

To answer your question, It depends a lot on the hardware you're using.

For comsumer/prosumer level hardware, I've read and agree with a general rule that for RAID5, 4 drives or less is desirable. Anything 5 or more you want RAID6.

This is because statistically as you rebuild a degraded RAID5 array, the changes of encountering an unrecoverable read error [URE] happen more frequently.

Failure is always "inevitable" over time. RAID is an extra layer of complexity that can make it easier, or more difficult to recover from failure.
 
A more important question is what are the size of the drives you'll be using. I'm guessing 2TB or larger. In that case you almost certainly want to use RAID-6. With the rebuild time on an array with large drives you are at a much greater risk for a 2nd drive failure and therefore total array loss. With the low cost of drives there's almost no reason not to go with RAID-6 or RAID-Z2 or even Z3.
 
I'm evaluating my new storage options while my current server fills up, and have to make a decision. I'd like to build a system with about 9 drives in the pool but I can't decide what level of hardware RAID I should use. Since I'll have more than the necessary numbers for RAID 5 (3) and RAID 6 (4), I have to decide if losing a second drive to parity would be advantageous.

Statistically, with what number of drives does RAID 6 become beneficial? Additionally, is there a point where a failure is all but inevitable?

At work (with proper backups on tape) I use RAID6 for 6 to 12 or so drives. More than that I will move to RAID60 or separate RAID6 arrays as I have no need for 20TB+ filesystems.

At home where all storage is on linux (meaning I have good software raid available) I choose to use individual 5 or 6 individual 2TB disks (that the individual SMART raw values are automatically watched for abnormalities) and backup the small fraction of data that is essential.

John
 
At work (with proper backups on tape) I use RAID6 for 6 to 12 or so drives. More than that I will move to RAID60 or separate RAID6 arrays as I have no need for 20TB+ filesystems.

At home where all storage is on linux (meaning I have good software raid available) I choose to use individual 5 or 6 individual 2TB disks (that the individual SMART raw values are automatically watched for abnormalities) and backup the small fraction of data that is essential.

John

I would agree with this ^.
 
With the rebuild time on an array with large drives you are at a much greater risk for a 2nd drive failure and therefore total array loss.

On a good raid (for example linux software raid) it takes 8 to 10 hours to rebuild a 10 to 12 disk 2TB raid6. Although with the current failure rates of drives I think its crazy to use raid5 for that. Then again with a proper tape backup I guess you can get away with the downtime.
 
The short versions:
http://www.zdnet.com/blog/storage/why-raid-5-stops-working-in-2009/162
http://www.zdnet.com/blog/storage/why-raid-6-stops-working-in-2019/805

The long version:
http://queue.acm.org/detail.cfm?id=1670144

A snippet:

Consider:

Long rebuild times. As disk capacity grows, so do rebuild times. 7200 RPM full drive writes average about 115 MB/sec - they slow down as they fill up - which means about 5 hours minimum to rebuild a failed drive. But most arrays can’t afford the overhead of a top speed rebuild, so rebuild times are usually 2-5x that.

More latent errors. Enterprise arrays employ background disk-scrubbing to find and correct disk errors before they bite. But as disk capapcities increase scrubbing takes longer. In a large array a disk might go for months between scrubs, meaning more errors on rebuild.

Disk failure correlation. RAID proponents assumed that disk failures are independent events, but long experience has shown this is not the case: 1 drive failure means another is much more likely.
 
Thanks everybody for the links and comments.

Like many others on [H] I'm just using this as a vault for anything I want to share between systems: media, disk backups, and software archives. I'm currently using an LVM solution and don't like that when I add drives, I still have a single point of failure. My design parameters for the replacement is to be able to support data storage for some time in the future (I want to fill my case with drives at the beginning and not add to it again).

Much of that data isn't mission-critical, but I would really hate to have to rebuild it all again. Maybe FlexRAID is an alternative?
 
FlexRAID I believe is now charging for their software. Depending on your level of technical ability and because you aren't looking to grow your array one drive at a time you should check out ZFS. That also gives you the option of RAID-6 level or greater parity disks and a host of other awesome features not available with standard RAID.
 
FlexRAID I believe is now charging for their software. Depending on your level of technical ability and because you aren't looking to grow your array one drive at a time you should check out ZFS. That also gives you the option of RAID-6 level or greater parity disks and a host of other awesome features not available with standard RAID.

I checked out ZFS further, and that sounds more suitable for me. I like that it's more flexible in terms of adding drives and takes care of drive pools and file systems.
 
I checked out ZFS further, and that sounds more suitable for me. I like that it's more flexible in terms of adding drives and takes care of drive pools and file systems.

The biggest problem I have with ZFS besides having to run it on a dedicated system is the lack of dynamic pool expansion. You can't just add a single drive to an existing pool and gain it as usable space, you have to add one or more parity drives along with it. Whereas with Flexraid or hardware raid you can add a single drive to an existing say 4-drive RAID6 and gain the entire drive's worth of usable space. With ZFS your only choice is to backup and reformat the pool or waste drives on parity drives with every pool expansion. In essence what ZFS does when expanding a pool is really just array spanning (chaining multiple arrays together) and calling it one pool. Not very efficient.

Not to mention, ZFS is a striping parity system meaning all the drives have to be spinning to access any data, whereas with systems like Flexraid or unRAID (blech) only the disk with the data you're interested in needs to spin up while the rest can sleep. Striping creates extra risk in the context of archival/media storage. In fairness to ZFS though its checksumming and protection against bit corruption is pretty unparalleled. However that may not be an exclusive feature forever as the Flexraid developer is finishing an "NZFS" implementation which will by all indications usher in ZFS-like data protection features and sit on top of windows.

Do a bit more research before jumping in. Personally I've been a hardware RAID believer for years and thats finally coming to an end as I'm migrating about 120TB to Flexraid. It all comes down to usage scenario: for enterprise, hardware RAID and more specifically NAS appliances are defacto whereas most of us enthusiasts on forums are interested in storing our growing home media collections and for that, the low cost software based parity systems shine.
 
Last edited:
Yeah, I would say only to use hardware raid in an enterprise situation. For home, I would stick to mirrors, individual drives, or something like unraid/flexraid. If you know what you are doing, ZFS can be a good option (I am using it currently).

With big drives (1TB+) DEFINITELY go with a more-than-one drive failure capability. I am running RaidZ2 (kind of like Raid6) plus an online hotspare, and then I have another matching drive sitting in my closet ready to install if(when) any of my drives do fail.
 
With big drives (1TB+) DEFINITELY go with a more-than-one drive failure capability. I am running RaidZ2 (kind of like Raid6) plus an online hotspare, and then I have another matching drive sitting in my closet ready to install if(when) any of my drives do fail.

Slightly off topic, but why raidz2 with a hot spare instead of raidz3, which is the same amount of disks but can survive a 3 disk failure even without the time to rebuild the hot spare.
 
SnapRAID might be a bit more challenging for the user that just wants to set something up quick and begin storing data, since its commandline-only and the third party attempt at a GUI doesn't seem to be updated as frequently as snapraid -- potential for compatibility issues isn't something that would help me sleep easy when it comes to data protection software. Plus its snapshot-only with no realtime, does not pool drives, and maxes out at 2 parity units (FlexRAID supports up to 100 in snapshot mode). If anything that comparison chart seems like an unintentional advertisement for Flexraid.

I think its another lesson in you get what you pay for but by all means people should try everything and see what works best for them. In fairness to SnapRAID, 2 parity units max is probably good enough for most users, so it comes down to whether you can benefit from Flexraid's realtime RAID function, appreciate that the GUI is written by the same developer and the software is more mature by several years. When you consider that your data may be worth thousands or tens of thousands then basing the decision of what software is going to protect that data on whether its free or $50 may not be something you want to cut a corner on.
 
Last edited:
SnapRAID might be a bit more challenging for the user that just wants to set something up quick and begin storing data,

Actually, snapraid is quicker and easier to set up than flexraid. All you have to do is edit a few lines in a configuration file, then type "snapraid sync". Flexraid takes much longer, and is more error-prone to set up.

And snapraid is open-source and GPL, so it is even better than flexraid for dependability. If the developer of snapraid disappears, snapraid will live on.
 
Does SnapRAID do drive pooling? That feature seems to be completely absent from the comparison table which would mean its skewed to its strengths while omitting its weaknesses -- not an apples to apples comparison. So far my impression of Snapraid is that it's an updated version of disParity.

In any case drive pooling is another major consideration and AFAIK part of the Flexraid developer's overhead since pooling is where things get tricky as it requires API and signed drivers to be able to interact at the O/S and filesystem level, meaning licensing costs.
 
Last edited:
Does SnapRAID do drive pooling? That feature seems to be completely absent from the comparison table. If not that's another major consideration. So far my impression of Snapraid is that it's an updated version of disParity.

No, snapraid does snapshot RAID. If you want drive pooling, there are many other options. I like to choose my tools a la carte, so I can choose the best tool for each job (or not, if I don't want the job), rather than have several mediocre tools bundled together. For my usage, I don't need drive pooling. When I rip my movies, I rip them to whatever HDD is not yet full. When I watch movies, MediaBrowser takes care of aggregating them together into one list.

I wouldn't call snapraid an "updated version of disparity". One thing snapraid has going for it is it utilizes the open-source parity code used by mdadm (linux software RAID), which is highly tested and optimized. snapraid is significantly faster than flexraid at computing parity (and also checksums, I think).
 
I use SnapRAID over FlexRAID - I personally found it easier to setup than SnapRAID. My test failure / restore also worked unlike my FlexRAID attempt, although that was probably due to my installation screw-ups than anything. I don't give a smeg whether it is open source or not, I just want it to work. There is no guarantee that if an open source developer leaves another one will pick up where they left off. So far it has worked so I've been happy with it.

I might test out FlexRAID again as an extra set of parity drives, to guard against the failure of any one product. I guess that shows another benefit of the abstracted snapshot approach. I actually prefer copying data over via native tootlsets in any case - I trust MS more than the makes of either product so copying files over to a pooled drive is not on my list of benefits.

The comparison table may be an advert for other systems in that it is actually factual, which is quite refreshing to a degree. It shows FlexRAID and ZFS in a better light than SnapRAID for some areas - so be it, happens to be true.

I use Media Browser for aggregation as well.
 
I use SnapRAID over FlexRAID - I personally found it easier to setup than SnapRAID.

Is there a typo here? Did you mean you found SnapRAID easier to setup than FlexRAID?

I certainly found snapraid easier to set up than flexraid. Snapraid comes with some example configuration files, and it takes only a few minutes to change a few lines in the templates, then done! Just type "snapraid sync" and you are off to the races.
 
Is there a typo here? Did you mean you found SnapRAID easier to setup than FlexRAID?

I certainly found snapraid easier to set up than flexraid. Snapraid comes with some example configuration files, and it takes only a few minutes to change a few lines in the templates, then done! Just type "snapraid sync" and you are off to the races.

Yeah, you are right, I meant the other way around. I am a fan of GUI's in general though, trouble with command lines is it is very easy to mistype things sometimes, or make a change to a config file that you didn't mean, and then its your data off to the races :)
 
with command lines is it is very easy to mistype things sometimes, or make a change to a config file that you didn't mean, and then its your data off to the races :)

With snapraid, there are very few things you can mistype that will destroy your data. I cannot think of anything that is just a few characters off from causing data loss. I think that comes from snapraid being well designed, but also from the way snapshot RAID works -- it never changes your data unless you are trying to restore something. Under normal operation, all it does is read your data, and write out parity and checksum data.
 
With snapraid, there are very few things you can mistype that will destroy your data. I cannot think of anything that is just a few characters off from causing data loss. I think that comes from snapraid being well designed, but also from the way snapshot RAID works -- it never changes your data unless you are trying to restore something. Under normal operation, all it does is read your data, and write out parity and checksum data.

You're probably right but I can quite quickly screw up my config by an incorrect configuration change in the config file, something that in theory would be harder to do in a GUI operation, since I assume like you I use a simple text editor that has no conception of the intended use. So no data loss but the potential of data loss goes up. GUI's generally provide a superior level of feedback and control.
 
You're probably right but I can quite quickly screw up my config by an incorrect configuration change in the config file, something that in theory would be harder to do in a GUI operation, since I assume like you I use a simple text editor that has no conception of the intended use. So no data loss but the potential of data loss goes up.

That seems to be more of an irrational fear than a reasonable problem. You can screw things up with a GUI if the system does not do much error-checking of inputs, and a text configuration can be very robust if the program does extensive error-checking on the inputs. Or vice versa. Such problems are not a function of GUI or text, but rather of good design or bad design.
 
Yes you *can* screw things up using anything but I haven't seen many CLI's giving context help, URL's to FAQ's interactive assistance lately. SnapRAID I suspect does nothing in that area - nor I suspect does it error check the contents of the configuration file other than to confirm that the commands are valid. No idea if the GUI does btw.

Personally, I find GUI's easier to use than CLI but each to their own. I suspect as a general rule most people do otherwise we'd still be using Edlin / Vi for word processing.
 
SnapRAID I suspect does nothing in that area - nor I suspect does it error check the contents of the configuration file other than to confirm that the commands are valid.

Your suspicions are incorrect.
 
I will-be/am using 5 drives raid 6, but they are 3TB drives and I'd rather not have to screw with it when I add more.
 
I will-be/am using 5 drives raid 6, but they are 3TB drives and I'd rather not have to screw with it when I add more.

I aggree, best to have it set in the beginning, although going from raid 5 to 6 wasn't so bad thanks to the help of Jus and others...(just seemed like a pain at the time).
 
Back
Top