RAID frustration

[jojo69]

OK, got an x58 Gigabyte ud3 here, had the system (win7) on a RAID 1 of .25TB SATA1 WD blacks.

I come home from a trip and the "Intel(R) Matrix Storage Console" is telling me I have a degraded RAID due to a failed drive. I'm like oh shit and rush out and buy a new set of SATA2 drives. I set the new drives up in RAID 1 and clone the system over no problem, get the system booting off that, good deal.

I plug the old drives into another box individually to see what is going on and there is no SMART problem on either of them, in fact they both pass the normal tests with flying colors, they have both been operating in other roles for weeks now flawlessly.

I come downstairs this morning and "Intel(R) Matrix Storage Console" is telling me I have a degraded RAID due to a failed drive...my brand new drives. I'm like "bullshit, you were lying to me last time" and set the "failed" drives status back to normal and hit "rebuild array" the recovery goes to 99.99 percent and then tells me it can't, that it has a bad drive...but it is the OTHER drive this time. I'm like yeah right, pull the drives out, check SMART on another box, they are both fine...WTF?

I have tried rebuilding the thing 3 times now, it always goes 99.99% and fail, "failed" drive seems random.

Pretty frustrated right now. If RAID configurations are more fragile than the physical media in them what the hell good are they?

 

[Thermonuclear]

Are you ONLY checking SMART? Or are you running some other sort of test? Or multiple others, preferrably...?

 

[jojo69]

first batch I ran a whole suite of tools

this new batch, it can't even decide which one is bad and these came out of their sealed antistatic 2 weeks ago, so with the results from the last batch I just checked SMART

 

[Thermonuclear]

Well, I've seen plenty of drives bad out of the box, with no previous use at all. I'd run a full check anyway. Normally I run a MFG test. Some people prefer 3rd party however.

 

[jojo69]

yeah, but both of them? and after rejecting good drives 2 weeks ago?

starting to think this RAID stuff is for the birds

 

[Thermonuclear]

RAID can be a pain, yes. I haven't used it in a while. Move the RAID partition to a separate drive and run them individually?

 

[jojo69]

starting to think so, do manual backups, less bang head on desk than this

 

[schmuckley]

starting to think so, do manual backups, less bang head on desk than this

^^ This

 

[m1abram]

yeah, but both of them? and after rejecting good drives 2 weeks ago?

starting to think this RAID stuff is for the birds

For home use with consumer raid controllers you are very correct. RAID in that situation has no benefit while having many downsides.

For enterprise it is a whole different game.

 

[brutalizer]

If you used ZFS and ZFS behaved like this, then the standard procedure is to check the hardware. For isntance, the PSU if it is faulty, HBA, etc. From time to time, ZFS users post threads like this, and it is always the hardware that is faulty, shaky PSU, etc.

So, I would check the rest of the hardware. Just a suggestion.

 

[mwarps]

Check your RAM. I had crap ram in a box that had an Intel RAID1, and it degraded weekly. It always managed to somehow recover. Changed the RAM, no more badness.

 

[Lost-Benji]

If you used ZFS and ZFS behaved like this, then the standard procedure is to check the hardware. For isntance, the PSU if it is faulty, HBA, etc. From time to time, ZFS users post threads like this, and it is always the hardware that is faulty, shaky PSU, etc.

So, I would check the rest of the hardware. Just a suggestion.

This is the first response that actually gets to a point.

OK, got an x58 Gigabyte ud3 here, had the system (win7) on a RAID 1 of .25TB SATA1 WD blacks.

I come home from a trip and the "Intel(R) Matrix Storage Console" is telling me I have a degraded RAID due to a failed drive. I'm like oh shit and rush out and buy a new set of SATA2 drives. I set the new drives up in RAID 1 and clone the system over no problem, get the system booting off that, good deal.

I plug the old drives into another box individually to see what is going on and there is no SMART problem on either of them, in fact they both pass the normal tests with flying colors, they have both been operating in other roles for weeks now flawlessly.

1. What are you using for the Intel, Intel Matrix or Intel RST software and what revision of drivers?
2. What ports are the drives on?
3. The ".25TB" (0.25TB or for the rest of world, 250GB) are showing their age.
4. Have you replaced the cables?
5. Have you tested with a different PSU?
6. Have you made sure that your system it's likely years of bloat-ware are not to blame?

Pretty frustrated right now. If RAID configurations are more fragile than the physical media in them what the hell good are they?

RAID is extremely simple to most who take the time to understand it. People who fear it or bag it are either smart enough to know they don't understand it and steer clear or there are those who think they know and have biased opinions. Not a shot at anyone, just my 2 cents worth.

For future responses:
The board you have is packing an ICH10R, these should support RST drivers/software but as Intel has had a poor run with getting drivers smooth and bug free, combined with your choice of old drives and now new drives in nothing more than RAID-1; you should be running the Intel Matrix version 8.9 drivers/software.

As you have a Gigabyte board, make sure you DO NOT have the shitty GB-SATA IDE RAID drivers/software installed. This is shit software and a big culprit to system slow-downs and shitty I/O.

If you wish to get to bottom of this, look at your system and it's OS/software load. I feel the issues is likely caused by the OS/software/bloatware rather than drives being faulty. Please post a screen-shot of MSCONFIG start-ups (you will need a couple to get all).

I am NOT ruling out a dud ICH either.

Another option that should have already been considered; Turn off the ICH RAID and set to AHCI. Put OS on single, fresh drive and get working. Add second matching drive and then enter DISK MANAGEMENT. "Add Mirror" to the hidden boot partition then do the same for the OS partition. You will have a sluggish system for a hour or so while they sync but once done, single drive write speeds and up to double drive read speeds and totally hardware independent RAID-1.

 

[m1abram]

Let me repharse what I meant above. RAID is not for home if you do not want to spend the time to understand its ins and outs. I run a rather large ZFS setup at home and manage many large RAID arrays at work, however I would never tell my dad to setup a RAID array at home. No reason to.

Like the OP my dad needs backups and as everyone will agree RAID != backups. So if you are using a RAID1 thinking you are doing it to avoid backups, dont. Get a proper backup setup like Crashplan or the like for your home, much simpler setup and will function for you better than your RAID 1.

 

[Deadjasper]

I gave up on RAID completely. You would think that writing the same data to 2 different HD's would be a no brainer but apparently it's not. I've had so many issues with RAID that I threw in the towel and just image my data periodically.

 

[m1abram]

I gave up on RAID completely. You would think that writing the same data to 2 different HD's would be a no brainer but apparently it's not. I've had so many issues with RAID that I threw in the towel and just image my data periodically.

Again if what you want is backups you should never have been using RAID in the first place. It is not backup.

 

[jojo69]

Again if what you want is backups you should never have been using RAID in the first place. It is not backup.

yeah, yeah, we get the mantra, the question is...if it is not a backup WTH is it? Or put another way; if data is MORE fragile in a RAID1 than not what is the actual point?

 

[m1abram]

yeah, yeah, we get the mantra, the question is...if it is not a backup WTH is it? Or put another way; if data is MORE fragile in a RAID1 than not what is the actual point?

It is for high availability, and in some cases higher performance. It is never used as a backup solution.

Consumer RAID is fragile because you are using drives not designed for RAID and are actually designed in a way the causes most RAID issue. Other reason it appears fragile is consumer gear is fragile. And lastly it appears fragile due to lack of understanding.

 

[staticlag]

yeah, yeah, we get the mantra, the question is...if it is not a backup WTH is it? Or put another way; if data is MORE fragile in a RAID1 than not what is the actual point?

The intel controller is the problem.

Simply create a software RAID1 in windows and you will have a very easy to use, portable, plug and play RAID set (you can plug the drives into any windows install together and it will recognize the RAID) that performs as good as or better than the intel RAID

 

[Deadjasper]

I tried Windows soft RAID and the drives kept getting out of sync. When I switched to ReFS the situation got even worst, they would still get out of sync but would not re sync again. At the consumer level, where most of us are, RAID is just a bad idea.

 

[m1abram]

I tried Windows soft RAID and the drives kept getting out of sync. When I switched to ReFS the situation got even worst, they would still get out of sync but would not re sync again. At the consumer level, where most of us are, RAID is just a bad idea.

Yep and of little use to most consumers since high availability is not that big of a concern for most consumers. Do proper backups and you are good.

Course if you want to run a large SAN at home you will of course probably want RAID, in which case I recommend getting a proper RAID controller or better yet look into using ZFS (which is what I use).

 

[uOpt]

Do you use a SMART tool that gives you the log of errors? The last 5 errors are logged in the drive.

It looks like you have a board issue (bad controller) or maybe bad power. Also, your board doesn't have ECC so if there is a RAM module flipping bits that would show up as an error and the computer wouldn't know why.

I would never use that kind of onboard RAID. My tools of choice are Linux md or FreeBSD ZFS. They tell you specifically why they got upset about a drive and kicked it out.

 

[Gonzo_SK1]

WD Blacks are desktop drives without TLER.

This is contributing to your problem.

Edited to add:

The issue is drives without TLER do not let "RAID be RAID"

Here's what happens.

Consumer desktop drives encountering a read error (inevitable on magnetic storage) will retry, and perform error recovery routines that can take up to a couple minutes. They stop communicating to the controller while doing so. The controller thinks "dead drive" and pulls the drive from the array.

TLER times out error recovery in less than 7 seconds and promptly communicates to the controller "hey, I have a bad sector" -- The RAID controller than gets to do its thing... it says "no worries, that sectors' mirror (or parity in RAID 5) is intact, so we'll recover the data from the intact part of the array, take the affected sector out of operation and keep chugging on.

Without TLER the drive gets marked as dead and pulled off the array. Then, during rebuild, every single sector on both drives are read and/orwritten so if there is an error on the other drive (and a long non-TLER recovery process causing the drive to be pulled off the array) you have a rebuild failure. But most likely its not the same sector on each drive getting read errors. And most likely the errors are recoverable. So if you had TLER, the RAID controller would simply do its thing.

Running RAID with consumer desktop drives is dangerous, especially RAID 5 because of this issue. (Very probable that a different drive in the array will have a timeout drop out during a rebuild causing the system to treat the entire array as gone).

This is the reason your drives seem to be working fine when pulled from the array and used as singles elsewhere. Desktop controllers will sit and wait patiently for the normal desktop time length error recovery routines.

I recommend WD RED or RE style drives.

And yes, I speak from experience with this very issue.

 

[jojo69]

thanks Gonzo, I think that is the best answer yet

 

[Deadjasper]

Yeppers, certainly mirrors my experience. (pun intended )

 

[uOpt]

WD Blacks are desktop drives without TLER.

This is contributing to your problem.

Edited to add:

The issue is drives without TLER do not let "RAID be RAID"

Here's what happens.

Consumer desktop drives encountering a read error (inevitable on magnetic storage) will retry, and perform error recovery routines that can take up to a couple minutes. They stop communicating to the controller while doing so. The controller thinks "dead drive" and pulls the drive from the array.

TLER times out error recovery in less than 7 seconds and promptly communicates to the controller "hey, I have a bad sector" -- The RAID controller than gets to do its thing... it says "no worries, that sectors' mirror (or parity in RAID 5) is intact, so we'll recover the data from the intact part of the array, take the affected sector out of operation and keep chugging on.

Without TLER the drive gets marked as dead and pulled off the array. Then, during rebuild, every single sector on both drives are read and/orwritten so if there is an error on the other drive (and a long non-TLER recovery process causing the drive to be pulled off the array) you have a rebuild failure. But most likely its not the same sector on each drive getting read errors. And most likely the errors are recoverable. So if you had TLER, the RAID controller would simply do its thing.

Running RAID with consumer desktop drives is dangerous, especially RAID 5 because of this issue. (Very probable that a different drive in the array will have a timeout drop out during a rebuild causing the system to treat the entire array as gone).

This is the reason your drives seem to be working fine when pulled from the array and used as singles elsewhere. Desktop controllers will sit and wait patiently for the normal desktop time length error recovery routines.

I recommend WD RED or RE style drives.

And yes, I speak from experience with this very issue.

This isn't as black and white.

Sometimes you want you RAID software to wait for the drive to try hard to recover a block.

I had that situation just Friday when the array was degraded for other reasons and the recovery (the sync with a new drive) hit a bad block that couldn't be read. In this case the block you can't read can't be constructed alternatively from the other drives, so if the read doesn't succeed you fail another drive. In that situation you really want the drive to go all out for extended periods of time.

I also fail to see why this minor change in drive firmware policy warrants the much higher price.

 

[Gonzo_SK1]

This isn't as black and white.

Sometimes you want you RAID software to wait for the drive to try hard to recover a block.

I had that situation just Friday when the array was degraded for other reasons and the recovery (the sync with a new drive) hit a bad block that couldn't be read. In this case the block you can't read can't be constructed alternatively from the other drives, so if the read doesn't succeed you fail another drive. In that situation you really want the drive to go all out for extended periods of time.

I also fail to see why this minor change in drive firmware policy warrants the much higher price.

You're right that its not black and white. And the situation you describe is why DELL, et. al., now no longer recommend RAID 5 in particular for very large drives. With published specs guarantying at least 1 non-recoverable read error out of every 10^14 bits read, you've got maybe a 15 to 20% chance of an error during a rebuild using 4+ 3TB drives. RAID 6 can help with its second pool of parity information, but performance suffers and as drives get bigger and bigger we'll eventually hit this practical deploy-ability limit with that sort of array as well.

My post in this thread, however, was limited to the open poster's descriptions of the problems, which suggests to me that lacking / missing TLER is the issue.

As for the price of RED and RE style drives, I agree its unfortunate. And there is some evidence hard drive manufacturers do this on purpose (WD taking away the ability to activate TLER for consumer drives via an outside utility suggests some forcing in terms of trying to maintain that price differential -- though there are other benefits, lower heat for example, in the RED and RE lines).

The good news is that there are GREAT deals in the 1TB to 2TB range of drives if one limits themselves to SATA2 interfaces. For sequential reads larger than the buffer, SATA2 is more than enough to carry the maximum physical transfer rate of the platters and then same, and there are some deals out there. I just picked up a half dozen 1TB RE3 drives for less than $50 each through an Amazon serviced dealer, for example.

Ultimately, newer protection and redundancy schemes will have to fill the price gap for a period of time until very large capacity SSD drives hit the market. Expect advancements, as well, where a single SSD "device" has the equivalent of RAID 6 double parity diagonal parity striping built in (i.e., 8 TB of NAND ships. 5TB of space) and new RAID levels and controllers designed to assure mirror style redundancy on these on a device to device level.

Magnetic storage has long been a bottle neck in PC performance anyway, but as the mathematics of error rates as the single devices get larger are this is going to put traditional RAID thinking on the path toward extinction in my view.

 

[sub.mesa]

TLER is no more than a workaround for faulty RAID engines. There is no reason a good RAID5 engine would need TLER. ZFS RAID-Z is one example; and the differences between RAID-Z and RAID5 are irrelevant to the harddrive.

Upon reading an unreadable sector on a harddrive, the harddrive would trigger recovery which can last up to 120 seconds. Unless, you overwrite the affected sector. That is exactly what ZFS does. Read redundant information, write to the disk that has a timeout or read error. Simple enough.

The problem is that crappy RAID engines cannot differentiate between a disk doing recovery, and a failed disk. That is a design limitation, not the fault of RAID5 or the lack of TLER on the harddrive side.

TLER is actually dangerous, as uOpt already addressed, in cases where you lost your redundancy. So you do not want TLER unless you absolutely need it. The best, of course, is to avoid legacy RAID altogether and use a 3rd generation filesystem like ZFS that has much more intelligence to deal with these issues.

 

[evilsofa]

yeah, yeah, we get the mantra, the question is...if it is not a backup WTH is it? Or put another way; if data is MORE fragile in a RAID1 than not what is the actual point?

RAID 1 allows you to keep running a mission-critical task during a drive failure when you have only two disks available.

If your mission is relying on only two disks, then either the mission is not very critical, or you have very severe space or power constraints such as a laptop, or management is making very poor decisions. If you're using RAID 1 at home, you're doing it wrong.

 

[uOpt]

TLER is no more than a workaround for faulty RAID engines. There is no reason a good RAID5 engine would need TLER. ZFS RAID-Z is one example; and the differences between RAID-Z and RAID5 are irrelevant to the harddrive.

Upon reading an unreadable sector on a harddrive, the harddrive would trigger recovery which can last up to 120 seconds. Unless, you overwrite the affected sector. That is exactly what ZFS does. Read redundant information, write to the disk that has a timeout or read error. Simple enough.

The problem is that crappy RAID engines cannot differentiate between a disk doing recovery, and a failed disk. That is a design limitation, not the fault of RAID5 or the lack of TLER on the harddrive side.

TLER is actually dangerous, as uOpt already addressed, in cases where you lost your redundancy. So you do not want TLER unless you absolutely need it. The best, of course, is to avoid legacy RAID altogether and use a 3rd generation filesystem like ZFS that has much more intelligence to deal with these issues.

Very well put, thank you.

 

[uOpt]

yeah, yeah, we get the mantra, the question is...if it is not a backup WTH is it? Or put another way; if data is MORE fragile in a RAID1 than not what is the actual point?

It's not more fragile, just kind of useless for certain requirements.

Overall a system with raid1 and snapshots, combined with ECC memory to guard against slowly scrambling your data (including the snapshots) can perform reasonably well in the face of adversity, especially if you combine it with both scrubbing the RAID and a regular run of checksumming files you expect to be immutable. The major threat is losing the whole machine, e.g. a OCZ^H^H^HPSU blowing up all components.

You will, however, have to have good enough software and know it well enough to perform stunts like I had to with my array. As we have seen, it goes reasonably quickly, and unexpected even for people with longer experience, that the array goes offline. To then force-assemble it and gather your data bits with confidence about what has a risk to be corrupted isn't quite trivial. Naive controller software will make much of it impossible anyway.

 

[malcolmputer]

Since you switched to non-redundant non-RAID setup, make sure that your data is still there. Sometimes RAID confusions can actually corrupt your data.

I am just saying that you should check it before you over-write your backups.

 

[vorpel]

Excellent reply - thanks!

Raid is great as long as you use it for what it is intended with the appropriate hardware - that usually means too much $$$ for home/casual use. It is good to see/read more info on larger drives - definitely making me lean towards ZFS, FlexRaid, etc.

-dc

You're right that its not black and white. And the situation you describe is why DELL, et. al., now no longer recommend RAID 5 in particular for very large drives. With published specs guarantying at least 1 non-recoverable read error out of every 10^14 bits read, you've got maybe a 15 to 20% chance of an error during a rebuild using 4+ 3TB drives. RAID 6 can help with its second pool of parity information, but performance suffers and as drives get bigger and bigger we'll eventually hit this practical deploy-ability limit with that sort of array as well.

My post in this thread, however, was limited to the open poster's descriptions of the problems, which suggests to me that lacking / missing TLER is the issue.

As for the price of RED and RE style drives, I agree its unfortunate. And there is some evidence hard drive manufacturers do this on purpose (WD taking away the ability to activate TLER for consumer drives via an outside utility suggests some forcing in terms of trying to maintain that price differential -- though there are other benefits, lower heat for example, in the RED and RE lines).

The good news is that there are GREAT deals in the 1TB to 2TB range of drives if one limits themselves to SATA2 interfaces. For sequential reads larger than the buffer, SATA2 is more than enough to carry the maximum physical transfer rate of the platters and then same, and there are some deals out there. I just picked up a half dozen 1TB RE3 drives for less than $50 each through an Amazon serviced dealer, for example.

Ultimately, newer protection and redundancy schemes will have to fill the price gap for a period of time until very large capacity SSD drives hit the market. Expect advancements, as well, where a single SSD "device" has the equivalent of RAID 6 double parity diagonal parity striping built in (i.e., 8 TB of NAND ships. 5TB of space) and new RAID levels and controllers designed to assure mirror style redundancy on these on a device to device level.

Magnetic storage has long been a bottle neck in PC performance anyway, but as the mathematics of error rates as the single devices get larger are this is going to put traditional RAID thinking on the path toward extinction in my view.