"RAID's Days May Be Numbered"

[InvisiBill]

http://www.enterprisestorageforum.com/technology/features/article.php/3839636

The concept of parity-based RAID (levels 3, 5 and 6) is now pretty old in technological terms, and the technology's limitations will become pretty clear in the not-too-distant future — and are probably obvious to some users already. In my opinion, RAID-6 is a reliability Band Aid for RAID-5, and going from one parity drive to two is simply delaying the inevitable.

The bottom line is this: Disk density has increased far more than performance and hard error rates haven't changed much, creating much greater RAID rebuild times and a much higher risk of data loss. In short, it's a scenario that will eventually require a solution, if not a whole new way of storing and protecting data.

That means that in a perfect world using vendor specifications, we can expect one out of every 312 rebuilds to fail today with data loss. Moving to the next-generation 2 TB drives, this value drops to every 234 rebuilds, and with 4 TB drives it drops to every 117 rebuilds.

A lot of people here already know all this, but the article does a good job of showing the math behind the problem. Drives are getting more dense at a faster rate than improvements in reliability and speed are happening. This is increasing the chance of data loss, due to longer times where the array is unprotected.

 

[jimhsu]

This was already said somewhere, but:

1. Rebuilds generally don't catastrophically fail with a URE (unrecoverable read error) - if one occurs, that file simply contains an error.
2. If you do encounter a URE (e.g. parity fail), try the read again. A true URE (as opposed to physical/logical disk damage) should be independent of location on the drive.
3. Any data worth protecting hopefully has some sort of ECC.
4. You aren't relying on RAID as a backup, correct?

And to be clear, RAID was never intended to replace backups. If your controller fails, there goes your RAID array (typically).

 

[kharan5876]

If your controller fails, there goes your RAID array (typically).

Thats why I don't believe in hardware RAID.

 

[Kritter]

If your controller fails, there goes your RAID array (typically).

???

I thought that most controllers stored the volume information on the disks pre-MBR. And if the controller fails all you needed to do was replace it.

 

[kharan5876]

And if its no longer being manufactured you have to buy a used one off of ebay or something. Hopefully they didnt sneak in some kind of new firmware revision in some of the models that breaks compatability with your array,

 

[coo-coo-clocker]

This was already said somewhere, but:

1. Rebuilds generally don't catastrophically fail with a URE (unrecoverable read error) - if one occurs, that file simply contains an error.
2. If you do encounter a URE (e.g. parity fail), try the read again. A true URE (as opposed to physical/logical disk damage) should be independent of location on the drive.
3. Any data worth protecting hopefully has some sort of ECC.
4. You aren't relying on RAID as a backup, correct?

And to be clear, RAID was never intended to replace backups. If your controller fails, there goes your RAID array (typically).

All good until that last part. A controller failure doesn't necessarily imply anything about the data stored in the array.

 

[DD_nVidia]

RAID won't die out... They may come up with new more efficient types but we'll still have it.

Also, with SSD's replacing HDD's in the long term, the whole rebuild speed thing should be out the window.

And if the data is THAT important on the RAID array, then you'd be smart enough to buy a spare controller card, just incase.

 

[Ryan45]

And to be clear, RAID was never intended to replace backups. If your controller fails, there goes your RAID array (typically).

That is not true, we have had controllers die in our HP servers, if you replace with the same model the array will still be intact.

 

[Dan_D]

RAID controllers usually have configuration data stored on the drives as well as the controller. (LSI cards do anyway.) So you can replace a controller with a new one of that same brand, it should pick up the array. Some cards like the LSI's actually let you import foreign array types, but I am uncertain as to how well this works.

Both software and hardware RAID implementations have their drawbacks but generally speaking I'll take a good hardware RAID over software anytime.

 

[Zardoz]

Losing your controller does not mean you will lose the raid. There is a chance of that happening, but for the most part the new controllers will ether pickup the drives again or you will need to import the foreign configuration off the drives.

RAIDS are not just for redundancy they are also use for increasing IOPS. Where IOPS are concerned redundancy is just a extra.

 

[Tolyngee]

And if its no longer being manufactured you have to buy a used one off of ebay or something. Hopefully they didnt sneak in some kind of new firmware revision in some of the models that breaks compatability with your array,

Why is it that everyone keeps spare drives handy, but keeping spare controllers handy seems such a foreign concept?

 

[Justintoxicated]

old news, this is why these days people are no longer using raid 5 etc. You need at least a few extra drives with raid 6, not necessarily just 2. Also solid state is in the future, by the time raid-6 is no longer practical there will be new solutions I'm sure.

 

[unhappy_mage]

Why is it that everyone keeps spare drives handy, but keeping spare controllers handy seems such a foreign concept?

Because drives cost $100 and controllers cost $1000.

 

[Dan_D]

Why is it that everyone keeps spare drives handy, but keeping spare controllers handy seems such a foreign concept?

Well I think that is mostly a cost conscious decision. In the case of servers that have proper support contracts and are under warranty, a new controller is just a few hours a way free of charge. I think that's largely why they do things that way. As for spare drives, I can't tell you how many compaies run RAID5 arrays with no spare drives on hand. Most places I've worked seemed to lack that foresight no matter how much I told them they needed to do it.

It wasn't until I was proven right during a drive failure that one company got the message.

 

[unhappy_mage]

A single 8+2 rebuild with 1.5 TB drives reads and writes about 28.5 TB (read 1.5 TB*9 drives, write 1.5 TB*10 drives).

No, to rebuild an 8+2 array (which is really too big---try 6+2 sets max) you read 8 drives and write 1, for a total of 9*1.5=10.5TB transferred. Only if another drive fails midstream do you need to touch the ninth drive.

In my opinion, RAID-6 is a reliability Band Aid for RAID-5, and going from one parity drive to two is simply delaying the inevitable.

Your opinion is not backed by reality or mathematics. If you hit an URE while rebuilding a raid 5, you've just lost a sector; if you hit one while rebuilding a raid 6, you go read the other parity disk and keep going.

90 percent of full rate is in many cases a best-case number, and in many cases, full rebuild requires the whole drive.

Rebuilding requires the whole drive? What does that mean?

The bandwidth assumes that two channels of the fastest type are available to be used for the 9 or 10 (RAID-5 or RAID-6) drives. So for 2009, I am assuming two 800 MB/sec channels are available for rebuild for the 9 or 10 drives.

Except that in the hardware raid case, the controller has 300 MB/s symmetric full-duplex to each drive. The times go from 17.8k seconds to read down to 14.6k, and from 41.6k seconds to rebuild down to 14.6k again.

Device Hard Error Rate in Bits PB Equivalent
Consumer SATA 10E14 0.89
Enterprise SATA 10E15 8.88

Hmm. This looks like a good time to bring out my trusty calculator:
>>> 10.**15 / 8 / (1024 ** 4)
113.68683772161603
What?! 10 to the 15 bits is only 113 TB, not 8880!

In essence, the disk drive layer needs to have more knowledge of what is storage, or fixed RAID devices must be rethought.

Is English your first language?

The sooner they arrive, the better chance we have of avoiding MTDL.

I don't want to avoid Mean Time to Data Loss, I would rather calculate it. With, y'know, numbers.

This is definitely from an Enterprise Storage forum, all right. Numbers pulled out of a hat, incorrect assumptions, and FUD about disks. Heck, let's take a look at another article:

RAID controllers designed for hard disks and IOPS aren't going to cut it in the new world of solid state drives (SSDs) and pNFS, which are capable of feeding so much more data into the channel that streaming bandwidth is going to become the problem for RAID vendors.

Aha! RAID controllers are designed to do Input/Output operations Per Second, and SSDs have so much bandwidth that these IOPS will be insufficient for measuring performance, and we will have to replace them with controllers that focus on doing more... streaming? Dude, I don't even know where to begin. Reading a million 4k blocks from a million different locations is a substantially different problem from reading a sequential 4GB chunk, flash or no flash. If your controller can do lots of IOPS, it can do lots of IOPS from an SSD, too. But if you design your controller for sequential IO, you're screwed.

-1, Troll.

 

[thesmokingman]

Is the sky falling?

 

[Tolyngee]

Well I think that is mostly a cost conscious decision. In the case of servers that have proper support contracts and are under warranty, a new controller is just a few hours a way free of charge. I think that's largely why they do things that way. As for spare drives, I can't tell you how many compaies run RAID5 arrays with no spare drives on hand. Most places I've worked seemed to lack that foresight no matter how much I told them they needed to do it.

It wasn't until I was proven right during a drive failure that one company got the message.

Because drives cost $100 and controllers cost $1000.

Unfortunately neither of you read the statement I was responding to.

For personal use, and by the time it's no longer being manufactured, it's no longer $1K on Ebay. It's probably only $100, if not $10. So, it probably even costs less than the spare drives you have sitting around.

At that price, have a spare handy. Better than worrying whether or not you can find one when you finally do need it.

Your data's worthless if you don't have a controller to drive it.

 

[kharan5876]

Thats still $100 I have to waste on an old used controller for a one time recovery. By the time my SAS controller dies Id like to buy a new SAS2 controller or SAS3 or whatever is out by then.

Also I hope you have a nice big stockpile of disks to migrate your data to when you do have to build a new array.

Where is the real hardware RAID advantage? Performance? Arguably negligible, especially for us home users that just have basic media file servers for our home networks and not database servers processing thousands of requests a second.

Basically, I trust open source software RAID implementations with my data a heck of a lot more than some proprietary vendor crap.

 

[Keiichi]

This is why you go for a solution with dual controllers (hotswappable preferrably) in a +1 design.
This should mitigate downtime due to an array rebuild and add that extra 9 to uptime.

 

[C-rizzle]

Ok I'm not up to par on all othe fancier RAID methods.

I don't think those arguements apply to a simple Raid 1 set up.

I was about to set up a Raid 1 array to work as my "data back up plan".

Is that faulty thinking? Should I just leave them unraided and back up to the 2nd drive regularly ?

 

[jimhsu]

Re: Controller failure.

We are assuming that the controller is actually available for sale (as someone mentioned) when the RAID fails. Hoping that your obsolete controller is available for sale on Ebay is not my idea of redundancy. Unfortunately most RAID arrays don't consider duplicate controllers, despite it being as significant of a point of weakness as a drive failure during rebuild. Manual data recovery off something like RAID 5 or 6 is also possible, but I imagine that would get really expensive, really fast.

I think enterprises don't source obsolete components from Ebay. I may be wrong.

 

[Keiichi]

Ok I'm not up to par on all othe fancier RAID methods.

I don't think those arguements apply to a simple Raid 1 set up.

I was about to set up a Raid 1 array to work as my "data back up plan".

Is that faulty thinking? Should I just leave them unraided and back up to the 2nd drive regularly ?

1. Raid is not backup.
2. The point of backing up is to have a second set of data offsite. There really isn't any point of doing any backup if you're going to leave the second drive in the computer. You'll be better off doing a Raid1 array for protection.

 

[jimhsu]

The thing about RAID (e.g. RAID 1) as a backup plan is what if your computer goes on fire, killing off both drives? A good backup should be in more than one location -- for example, home, office, and a server in another state. Having all your backups in one location creates unpleasant correlations (e.g. a fire or flood that kills off one drive will most likely kill off the other).

Here's my current backup plan:
1. Weekly images to secondary hard drive (external)
2. Important documents checked out from SVN repository in remote server, and the repository itself duplicated on yet another server (see http://onlamp.com/pub/a/onlamp/2005/01/06/svn_homedir.html?page=last )
3. All other nonessential documents currently being encrypted and backed up with Backblaze (online backup service)

 

[C-rizzle]

Well, I wanted a "back up" of my data in case a drive failed. And I like the real time "back up" of Raid 1. Though I see what you mean, in case of fire or direct lightening strike or something killing both drives in the PC at once.

So I'll use an old harddrive in an external enclosure and put that in a fire proof safe as my fail safe "back up" plan.

 

[unhappy_mage]

Unfortunately neither of you read the statement I was responding to.

Actually, I read pretty well. kharan5876 suggested that a hardware-oriented strategy might have larger flaws than simply a single controller dying.

For personal use, and by the time it's no longer being manufactured, it's no longer $1K on Ebay. It's probably only $100, if not $10. So, it probably even costs less than the spare drives you have sitting around.

Suppose I bought a controller a couple years ago--an Areca 1160 (released August 2008 or so). It's about $700 on eBay now. I see a lot of people buying 16- and 24-port controllers these days; any reason to think they'll drop in price faster?

At that price, have a spare handy. Better than worrying whether or not you can find one when you finally do need it.

Why not have a spare motherboard, too? And maybe a power supply? At some point home users will (and should!) draw a line and stop spending more money to make things more available. I see no particular reason not to make "controller" the line.

Your data's worthless if you don't have a controller to drive it.

In summary, this is why I use software RAID. The software won't disappear, and it runs on many types of hardware.

 

[parityOCP]

@C-rizzle

Indeed, even RAID 1 is not a backup - think of it more like a real-time drive replication. Whatever happens on one drive (deletions, viruses, etc) happens on the other. It's designed to maximise up time, and in a case where the drives can be hot-swapped in and out, up time is maximised.

What people name as backup should be more properly referred to as "archival backup".

@Hardware vs Software RAID:

The arguments for and against will flow back and forth, from now until Armageddon, to be honest. Each has its advantages and disadvantages, but my own opinion is that as multi-core CPUs become faster, cheaper and more ubiquitous, and integrated file systems such as ZFS grow in popularity, software RAID will become more and more viable.

 

[Drudenhaus]

???

I thought that most controllers stored the volume information on the disks pre-MBR. And if the controller fails all you needed to do was replace it.

They do. "Lose your RAID card, lose your array" is total bullshit. Just replace the card and you're back to "up and running" status. Also, many cards support arrays created on multiple generations of hardware cards as long as it's the same manufacturer.

 

[Dan_D]

They do. "Lose your RAID card, lose your array" is total bullshit. Just replace the card and you're back to "up and running" status. Also, many cards support arrays created on multiple generations of hardware cards as long as it's the same manufacturer.

Exactly.

 

[Copyright]

I can also confirm after changing probably 30-40 RAID controller cards when I worked at HP's Data Center that it did not always mean array failure. Actually I can't think of any time it ever did.