RAID, URE's, Confusion.

[JaSauders]

(to cut to my direct question and bypass most of the background story, drop to TL : DR)

Good morning! First time poster here... I've continually dug up HardForum links on Google that have helped substantially but I never actually registered until now... better late than never, eh?

So I'm doing some research here on exactly where I want to go with my storage solution. I'd like to fire out an initial disclaimer: I'm not using RAID as a backup. The most important data in the array will be backed up to an external hard drive and taken off-site, likely to my office at work. I'll probably take it home monthly to refresh the data.

My home server is a custom build with an i3 3220T, 4GB of RAM, and a large ventilated case. It's running a headless OS - Ubuntu Server 12.04. I'm beyond satisfied with this build. Some of the services it runs would be: OwnCloud, Subsonic, Motion, Irssi, web, file, print, backup, 24/7 video surveillance (the cameras are POE cams that save to a samba directory), etc. It'll likely host media for my HTPC once the new drives are installed as well. In the past I've ran RAID 1 with 2x500, and it was beautiful, as I had two drive failures in three years on my old server build. I have since rebuilt my server but didn't upgrade storage yet, hence this thread.

If I use RAID, it'll be using the Linux software RAID "mdadm". I considered RAID 1, 5, 6, and 10. After comparing costs, pros, cons, etc., I've begun to lean more towards RAID 1 as I've used in the past. My only concern/confusion is I've read a ton about the risk of a URE, some of which has left me a little confused on exactly where the risk lies and what exactly happens in the event a URE pops up. Some information seems dated and others suggests that today a URE won't tank an array, it'll just result in a corrupt file that existed on that sector.

TL : DR​

Based on what I read, this is my understanding:

• RAID 5 (4 drives, 1 being parity) with 1 newly installed drive, a URE during the re-sync could tank the array.
• RAID 6 (4 drives, 2 being parity) with 1 newly installed drive, a URE during the re-sync won't matter due to there being two parity drives.
• RAID 6 (4 drives, 2 being parity) with 2 newly installed drives, a URE during the re-sync could tank the array.

And my big question... what about RAID 1?

• RAID 1 (2 drives, mirrored) with 1 newly installed drive, a URE during the re-sync..... tanks the array? Results in a sector of corrupt data? What are the consequences of a URE landing during a RAID 1 re-sync? All of the URE related info I can find is relating to RAID 5/6/10...

Appreciate the insight!

 

[patrickdk]

It is the same as raid5.

You have 1 disk of recovery, if that drive is replaced with a spare, a URE from the good one will tank that mirror, taking the array with it.

But all this being said, it seems this is very sadistic look at the issue.

All you need to do is pull out the URE drive, mirror it, once it's done, do a recover again, and you will only loose that 1 sector to 1 raid stripe in the worst case.

This is also what smart tests, and scrubs are for, to make sure there are no URE's.

 

[JaSauders]

It is the same as raid5.

You have 1 disk of recovery, if that drive is replaced with a spare, a URE from the good one will tank that mirror, taking the array with it.

But all this being said, it seems this is very sadistic look at the issue.

All you need to do is pull out the URE drive, mirror it, once it's done, do a recover again, and you will only loose that 1 sector to 1 raid stripe in the worst case.

This is also what smart tests, and scrubs are for, to make sure there are no URE's.

Thanks for your insight. What's ironic is I just had a conversation with a server tech who works with storage devices. I took the opportunity to run my question by him, but his answer was slightly different. He said if I was running a RAID 1 mirror and a drive failed, installed a new drive and it began the re-sync process, then a URE hit, it would only knock out that block of data, resulting in corrupt files for whatever was living there. He specifically said it would not take out the entire array nor would I have any data loss besides whatever was on that specific block that was unreadable.

I'm a little confused by your comment here:

All you need to do is pull out the URE drive, mirror it, once it's done, do a recover again, and you will only loose that 1 sector to 1 raid stripe in the worst case.

Pull the URE drive out and mirror it? It was already mirroring...? I apologize, but truth be told now I'm a little confused... Can I expect to lose my array if a URE lands during a RAID 1 re-sync? I just find that baffling... I can understand it not being readable, but to knock out an array due to one sector of unreadable data? Now that just blows my mind.

 

[patrickdk]

It depends on your assumption first. I was basing it on the same assumption of your raid5/6 examples. He wasn't.

If your drive hits a URE, and takes too long to respond, and the raid card/software knocks the whole drive out of the raid, THEN you loose the raid. If the raid card/software doesn't knock it out, you only loose that sector/stripe.

So we assume worst case, the raid knocks the drive out, cause of a timeout on the URE retry.

RAID 5 (4 drives, 1 being parity) with 1 newly installed drive, a URE during the re-sync could tank the array.
RAID 6 (4 drives, 2 being parity) with 1 newly installed drive, a URE during the re-sync won't matter due to there being two parity drives.
RAID 6 (4 drives, 2 being parity) with 2 newly installed drives, a URE during the re-sync could tank the array.
RAID 1 (4 drives, 2 mirrored) with 1 drive installed, a URE during resync could tank the array.

This I assume is why it says COULD tank, and not WILL TANK, cause the drive might not be knocked out of the raid.

 

[patrickdk]

Oh, this is why enterprise disks DONT retry URE's, and are raid friendly.

Also WD RED drives, and their TLER firmware option, all adjust this.

 

[JaSauders]

I see. Thank you for clarifying that. In the event that I hit a URE during a RAID 1 rebuild and my array "tanks" (as I've been calling it), does that mean presto - no more data? Or will the data still be accessible on that single drive that's still running? I was just gathering from that that if I get hit with a URE it's just game over, start back at the beginning, case closed. It was *that* in particular that was jaw dropping, because I found it hard to believe that one URE could tank your array and guarantee you hours worth of work/new rebuild time...

I was actually looking at getting WD Reds, so I suppose that's a good match. Do WD Reds come with the TLER option on by default? Or is that something I need to dig into and enable on my own?

I'm somewhat unfamiliar with TLER, but I gather that TLER is responsible for ensuring the drive moves along accordingly. AKA, if it's rebuilding and a URE lands, before the controller (mdadm in my case) has a chance to drop the drive, TLER kicks in and ensures that the drive keeps moving forward with the rebuild, thereby marking off the problematic sector as unusable. Eh? It sounds like TLER is a pretty substantial safety net against URE's when it comes time to rebuild your array.

I'm reading that TLER is a WD thing. I assume Seagate and other manufacturers have their equivalent to TLER that handles the exact same task?

I guess moral of the story is, use the proper tools for the job? AKA - RAID oriented HDDs for RAID implementations?

EDIT - Here's a funny comment from another forum with users talking about how having no TLER is a headache with RAID (WD Greens being the topic), and someone says "Just run the drives in RAID 1. Since there's no parity calculations, there's no reason for TLER." This contradicts what I've just read here... Perhaps the internet is a bad place for researching such a topic?

 

[tycoonbob]

Your Server Tech is correct.

A URE during a rebuild on a RAID 1 cannot tank the array. RAID 1 does NOT use parity in any way, shape, form, or fashion since it is a simple mirror. During a rebuild of a RAID 5/6/50/60/<Insert RAID number that uses parity>, the controller is using the parity and doing calculations to rebuild the missing data and place it on the new drive. A URE during this calculation will cause the second drive to drop from the array, and would take a RAID 5 critical.

TLER/CCTL is still used with a RAID 1.

 

[JaSauders]

Your Server Tech is correct.

A URE during a rebuild on a RAID 1 cannot tank the array. RAID 1 does NOT use parity in any way, shape, form, or fashion since it is a simple mirror. During a rebuild of a RAID 5/6/50/60/<Insert RAID number that uses parity>, the controller is using the parity and doing calculations to rebuild the missing data and place it on the new drive. A URE during this calculation will cause the second drive to drop from the array, and would take a RAID 5 critical.

TLER/CCTL is still used with a RAID 1.

But, in this particular case ---> "A URE during this calculation will cause the second drive to drop from the array, and would take a RAID 5 critical." if you have TLER-enabled-drives, wouldn't the drives be able to circumvent the URE and thereby continue rebuilding without dropping any drives?

That will give me some additional peace of mind with my mirror at home then (not that I'll run backups any less frequently - no worries there). I suppose the use of TLER in a RAID 1 just ensures the drive won't get dropped if it gets REALLY hung up, eh? But even if it does, that remaining drive should be readable I would think...

I guess that sums up the bulk of my confusion. For sake of ensuring I get "proper" drives, I'll stick to the WD Reds in this instance since I know they're TLER enabled. Thank you for your response - I appreciate it!

 

[tycoonbob]

TLER will cause the drives to go offline instead of hanging up on a read error. TLER is a time limit on how long a drive tries spends on a read error. If it cases the drive to go offline, there is a chance you could reboot the server and have the array come back online/healthy, but you could still have problems in the future with some corrupted data. A quality controller (in the case of hardware RAID) will do background scrubbing and ideally take care of UREs before they cause issues during a rebuild.

When looking for non-enterprise drives that have TLER or CCTL, don't forget the new Toshiba DT01ACA series (DT01ACA300 for 3TB). They are basically relabled Hitachi drives with CCTL, which is their version of ERC (Error Recovery Control).

 

[JaSauders]

TLER will cause the drives to go offline instead of hanging up on a read error. TLER is a time limit on how long a drive tries spends on a read error. If it cases the drive to go offline, there is a chance you could reboot the server and have the array come back online/healthy, but you could still have problems in the future with some corrupted data. A quality controller (in the case of hardware RAID) will do background scrubbing and ideally take care of UREs before they cause issues during a rebuild.

When looking for non-enterprise drives that have TLER or CCTL, don't forget the new Toshiba DT01ACA series (DT01ACA300 for 3TB). They are basically relabled Hitachi drives with CCTL, which is their version of ERC (Error Recovery Control).

Did you really mean "TLER will cause the drives to go offline", or did you mean TLER will cause the drives to label an unreadable block of data as "bad" and continue moving forward? TLER causing the drives to go offline sounds like the exact opposite of what I thought.

I understood that a URE would cause the HDD to repeatedly try reading that block of data, and due to the HDD continually trying to read that data, it would thereby become "unseen" by the controller, resulting in the controller dropping the drive from the array. I understood if anything was going "offline", it was due to the controller forcing it offline, and that TLER was to keep the drive moving forward so it doesn't get hung up on URE's and therefore it'll be constantly "seen" by the controller, resulting in no drops from the array.

...or am I totally off in left field?

 

[-Dragon-]

TLER doesn't cause the drive to go offline, it's the lack of TLER that causes the drive to go offline. TLER does what you thought and lets the rebuild continue EXCEPT you do lose that chunk of data.

 

[tycoonbob]

Yeah, my bad. Lots of distractions at work this afternoon, lol. TLER will tell the drive to skip it and move on instead of hanging up on it.

From the Wiki:
Modern hard drives feature an ability to recover from some read/write errors by internally remapping sectors and other forms of self test and recovery. The process for this can sometimes take several seconds or (under heavy usage) minutes, during which time the drive is unresponsive. RAID controllers are designed to recognise a drive which does not respond within a few seconds, and mark it as unreliable, indicating that it should be withdrawn from use and the array rebuilt from parity data. This is a long process, degrades performance, and if more drives fail under the resulting additional workload, it may be catastrophic.
If the drive itself is inherently reliable but has some bad sectors, then TLER and similar features prevent a disk from being unnecessarily marked as 'failed' by limiting the time spent on correcting detected errors before advising the array controller of a failed operation. The array controller can then handle the data recovery for the limited amount involved, rather than marking the entire drive as faulty.

 

[JaSauders]

Very nice, fellas. Thank you for your time and patience. I feel as though I understand TLER and UREs significantly more now. I feel confident in the redundancy vs cost ratio with getting two 3TB WD Reds to mirror in my server via mdadm. Should be a pretty decent combination.

 

[hotcrandel]

Just to be clear, RAID is for uptime, or performance, or possibly both, and sometimes flipped bit error checking.

It looks like you already know it doesn't take place of backups.

And the point I'm trying to make is RAID1 of data you have backups, the only advantage you have is uptime, and it might be better invested in a second backup.

 

[JaSauders]

Just to be clear, RAID is for uptime, or performance, or possibly both, and sometimes flipped bit error checking.

It looks like you already know it doesn't take place of backups.

And the point I'm trying to make is RAID1 of data you have backups, the only advantage you have is uptime, and it might be better invested in a second backup.

Preachin to the choir here, brother.

Having some degree of redundancy just makes more sense in my case. My server does a lot (I notated what it does in the first post) and I rely on it more than I realize. Since I ran a RAID mirror before (2x500) and had two failed HDDs over the course of a few years, it stands to good reason to continue some sort of an array. I really wanted to get some fat hard drive space in there, but since hard drives are available upwards of 4TB now (I'm aiming for 3TB) it helps negate my original desire to stack a bunch of disks together to build the volume to a very large level. Not to mention, 2x3TB is considerably cheaper than 4x2TB (WD Reds in both cases). Cheaper + still a sweet amount of space + mirrored redundancy... that's a win for me.

As far as backups go, I'm pretty confident that a 500GB drive should hold the most important stuff that I want backed up. The vast majority of what will be on this array is media, which I can re-extract from DVD if need be (even though that would be incredibly obnoxious), so things like that I won't back up. I happen to have two external enclosures, so my thought process is to take the 2x500 in the server currently and pop one drive in each.

One external will get nightly syncs, the other external will get monthly syncs. The monthly sync drive will live in my desk at work, which gives me a semi recent off-site backup, whereas the other drive will get nightly syncs in the event my computer somehow self destructs and the entire array and both HDDs are gone.

I think that sounds like a decent home brewed plan.