Is RAID5 safe with Five 2TB Hard Drives?

S

solaris54

2[H]4U
Joined
Jul 14, 2007
Messages
2,203
Im new to the hard drive enclosure and RAID 5.

I just setup a SansDigital TR5UT with Five 2TB hard drives. The purpose of this setup is to backup my collection of movie DVDs.

I thought I was set using RAID5 but now after reading some more on it Im worried that it might not be the best way to do it. From what Im reading even if you lose only 1 hard drive there could still be an error on one of the other drives which will make the whole array unuseable.

So at this point Im wondering if I should just do the Clean mode or JBOD where each hard drive shows up individually with no RAID?? I could just spread the movies over the 5 drives and if one fails I only lose that group.

I guess I could backup too with another set of drives?
 
Unlike you'll get an error. It's still a good option with that many drives and you have to keep in mind, a single error will do very little. It won't make the entire array unusable, you just get a tiny bit of corruption. RAID doesn't replace backups however.
 
RAID 5 is safe. RAID 6 is safer. RAID with no other backups of your data is useless and not safe.

RAID5 can survive a single drive failure. However, once you replace that drive, it has to be initialized. Depending on the controller and other things, this can take anywhere from 5-18 hours. During this time, all drives will be in constant use to re-create the failed drive. It is during this time that people worry that the rebuild would cause another drive near death to die, causing a complete array failure.

Doing a JBOD would be ok if you have backups. Personally I would hate to have to re-rip and re-encode 2TB of movies if I lost a single drive. Pulling them from backup would be slightly less time consuming.
 
I can't remember off of the top of my head but I think it is something like an 85-90% chance you can read a 5 drive array with no UBEs.
 
I think for 5 drives, RAID5 is fine. It is rare that you will have have 2 drive fail at the same time.

I am doing RAID6 with 8 drives.

RAID is absolutely better than no raid since you don't lose any data if 1 drive fails. But RAID doesn't replace backups. I also have a WHS system (currently 12TB) for backing up my important stuff. So my stuff exists in both the RAID6 and WHS.
 
Also these drives in RAID 5 are showing up at around 7.74 TB. What happened to the rest? Thought it would be closer to 10TB?

So as far as backup would it make sense to get another 5 drive enclosure?
 
solaris54 said:
Also these drives in RAID 5 are showing up at around 7.74 TB. What happened to the rest? Thought it would be closer to 10TB?

So as far as backup would it make sense to get another 5 drive enclosure?
Click to expand...

Each disk is just over 1.8TB or so and you have n-1 as was noted earlier.
 
Thanks for all the help.....Im kind of understanding now.

Last time I messed with RAID there was only 0 and 1. Never knew they came up with all these variations.
 
All these variations have been around for quite some time.

I remember setting up a RAID 5 array on a Compaq Proliant 1000 some 15 years ago using a Compaq Smart Array Controller (there was no numerical designation on the model number back then) and installing Banyan Vines on it. That was my first experience with an enterprise level server.
 
That's guy's a bit of a scaremonger to be honest. He may have a point with consumer drives, but the article is sensationalised to a certain degree. However, there are still a few outfits that won't go past 500GB/drive in an array (even with enterprise drives), simply to reduce the failure window during a rebuild.
 
Last edited:
Any redundant raid that most fits your needs is fine as long as you have good backups. Raid5 is a nice balance between performance and redundancy.

As for the size, I find this is a a pretty bad number fudging but all hard drive manufacturers do it. When they say 1TB, what they really mean is 1,000,000,000,000 bytes but 1TB is actually 1,099,511,627,776 bytes. So a 2TB drive is about 1.8TB and not 2TB. This shows even more when you get into multi 2TB systems.
 
adi said:
RAID5 can survive a single drive failure. However, once you replace that drive, it has to be initialized. Depending on the controller and other things, this can take anywhere from 5-18 hours. During this time, all drives will be in constant use to re-create the failed drive. It is during this time that people worry that the rebuild would cause another drive near death to die, causing a complete array failure.
Click to expand...
This isn't the only danger. The problem with 2TB disks, especially if they are not 4K sector disks, is that they have relative high BER rate for their capacity, so the likelyhood of BER actually occurring and translating into an unreadable sector is something to worry about.

If during a rebuild one of the remaining disks experiences BER, your rebuild stops and you may have headaches recovering from such a situation, depending on controller design and user interaction.

So i would say with modern high-BER drives you should say:
RAID5: 0 complete disk failures, BER covered
RAID6: 1 complete disk failure, BER covered

So essentially you'll lose one parity disk alone for the BER issue. Not everyone will agree with my analysis, but considering RAID5 with today's high-capacity drives 'safe' is open for debate.

RAID5 + a GOOD backup is something to consider, though.
 
No, Red Squirrel, that is incorrect.

One TB = 1,000,000,000,000 bytes. T is the standard metric system prefix meaning 10^12.

One TiB is 2^40 bytes = 1,099,511,627,776 bytes

Microsoft (in their OS), and many people sometimes write TB when they mean TiB. But you should always keep the correct definition in mind when you are actually counting bytes.
 
Yes. It is funny, however, that harddrive manufacturers still use KB/MB when dealing with cache size. If they were consistent, they should have used KiB/MiB here.
 
sub.mesa said:
If during a rebuild one of the remaining disks experiences BER, your rebuild stops and you may have headaches recovering from such a situation, depending on controller design and user interaction.
Click to expand...

I do not think this is correct. I think you are confusing a bit error (BER) with an unrecoverable sector read error.

An unrecoverable sector read error occurs when a sector simply cannot be read (perhaps it also includes cases when the ECC can detect the error but not correct it? not sure about that), even after the HDD goes into a special recovery cycle.

A bit error occurs when one (or more) bits are read back that are different than what was written to disk. It may occur because too many bits in a sector randomly flipped, and the ECC could not detect and correct the problem (the raw BER of the platter is much higher than 10^-14, but the built-in ECC can correct some bit flips).
 
Last edited:
So you're saying BER is the error count that 'escapes' the ECC correction? I do not believe that is correct, but i'm open to good arguments or links.

As i understand, the BER is what prompt bad sectors, where the number of errors exceed that of the ECC error correcting ability; and you will have an unrecoverable sector (Current Pending Sector in SMART output).

Also these links are interesting in this context:

http://blog.econtech.selfip.org/200...s-not-fully-readable-a-lawsuit-in-the-making/

The short story first: Your consumer level 1TB SATA drive has a 44% chance that it can be completely read without any error. If you run a RAID setup, this is really bad news because it may prevent rebuilding an array in the case of disk failure, making your RAID not so Redundant.
Click to expand...
Not sure on the numbers the article comes up with, though.

Also this one is interesting:
http://lefthandnetworks.typepad.com/virtual_view/2008/02/what-does-data.html

BER simply means that while reading your data from the disk drive you will get an average of one non-recoverable error in so many bits read, as specified by the manufacturer.
Click to expand...

Rebuilding the data on a replacement drive with most RAID algorithms requires that all the other data on the other drives be pristine and error free. If there is a single error in a single sector, then the data for the corresponding sector on the replacement drive cannot be reconstructed, and therefore the RAID rebuild fails and data is lost. The frequency of this disastrous occurrence is derived from the BER. Simple calculations will show that the chance of data loss due to BER is much greater than all other reasons combined.
Click to expand...

These links do suggest that BER works to produce un-recoverable sectors, and not 'escape' them as 'undetected' bad sectors, if i understood you correctly.
 
Are you claiming that it is impossible for a bit to be read back from a drive that is different from what was written?

Or are you saying that the term for that sort of error is not bit error rate?
 
The question I have for you is do you need to backup everything? Is some of this data replaceable? Is this home data or a business?

In my HTPC I have removed the raid and gone to single disks without any backup for the video because I could loose 2TB of recorded TV output on a disk and that would not be a huge loss. With that said I do have a real backup procedure and software for the 100 or so GB that I do not want to loose. For this I do monthly fulls, weekly differentials and daily incrementals to two different backup drives which I alternate from time to time.
 
drescherjm said:
The question I have for you is do you need to backup everything? Is some of this data replaceable? Is this home data or a business?

In my HTPC I have removed the raid and gone to single disks without any backup for the video because I could loose 2TB of recorded TV output on a disk and that would not be a huge loss. With that said I do have a real backup procedure and software for the 100 or so GB that I do not want to loose. For this I do monthly fulls, weekly differentials and daily incrementals to two different backup drives which I alternate from time to time.
Click to expand...

I am backing up my DVD collection and then plan on getting rid of the DVDs. It wouldnt be the end of the world if I lose it but a major inconvenience I guess.

Im thinking of just putting as much as I can on external or I could just buy another external enclosure like the Sans Digital that I have right now..........they arent that expensive.

What I was mentioning before was just getting rid of the RAID and going to the Clean mode and having the movies spread over 5 hard drives. If I lost one it wouldnt be that bad.
 
john4200 said:
No, Red Squirrel, that is incorrect.

One TB = 1,000,000,000,000 bytes. T is the standard metric system prefix meaning 10^12.

One TiB is 2^40 bytes = 1,099,511,627,776 bytes

Microsoft (in their OS), and many people sometimes write TB when they mean TiB. But you should always keep the correct definition in mind when you are actually counting bytes.
Click to expand...

Really? first time I hear of this. I knew about the metric, but thought in computing it was an exception. I still find it's a fudging of numbers though, most people and systems (ex: linux, windows) refer to a KB as 1024 bytes, and so on so it's deceiving when they use 1,000.
 
solaris54 said:
I am backing up my DVD collection and then plan on getting rid of the DVDs. It wouldnt be the end of the world if I lose it but a major inconvenience I guess.

Im thinking of just putting as much as I can on external or I could just buy another external enclosure like the Sans Digital that I have right now..........they arent that expensive.

What I was mentioning before was just getting rid of the RAID and going to the Clean mode and having the movies spread over 5 hard drives. If I lost one it wouldnt be that bad.
Click to expand...

I'd still go with raid 5, at least if you lose a drive you don't have to worry about referring to backups. It's also more convenient to have one large array then several small ones.

As for how to backup, personally I love HDD docks, they can plug in via usb or esata and you just plug a drive in, turn on the power, then run a backup script (ex: rsync). I have my backup script setup to mount by label, all my backup drives (only have two atm but want to get more) have the same label (ex: "backupdrive") and on each drive there is a symlink to a backup script that then backs up the proper data for that backup job. Each hard drive has it's own job. I can just add more hard drives into the pool anytime by making it use an existing job, or backup other data. Good idea to have at least two copies as hard drives can and do fail.

I only backup the most critical stuff like code, server configuration files like apache etc databases, and stuff of that sort. movies and stuff I don't bother but since I have raid5 the only way I lose them is if they get deleted or something. (accident, virus, etc).
 
I'm actually "toying" with doing something like a big RAID-Z3 pool for backup plus a bunch of smaller RAID-Z pools (3-4 drive) for the main storage.

The big thing here is I almost think these need to be real-time mirrored setups because I can't imagine copying 40TB+ of data without seeing an error or two from something if ever things went really really wrong.
 
parityOCP said:
That's guy's a bit of a scaremonger to be honest. He may have a point with consumer drives, but the article is sensationalised to a certain degree. However, there are still a few outfits that won't go past 500GB/drive in an array (even with enterprise drives), simply to reduce the failure window during a rebuild.
Click to expand...
Why is he a scaremonger? He is correct. Have you read his article? In fact, he has copied his argument from Adam Leventhal(?) that was one of the ZFS developers I believe.


Adam's argument goes likes this:
Disks are getting larger all the time, in fact, the storage increases exponentially. At the same time, the bandwidth is increasing not that fast - we are still at 100MB/sek even after decades. So, bandwidth has increased maybe 20x after decades. While storage has increased from 10MB to 3TB = 300.000 times.

The trend is clear. In the future when we have 10TB drives, they will not be much faster than today. This means, to repair an raid with 3TB disks today, will take several days, maybe even one week. With 10TB drives, it will take several weeks, maybe a month.

Repairing a raid stresses the other disks much, which means they can break too. Experienced sysadmins reports that this happens quite often during a repair. Maybe because those disks come from the same batch, they have the same weakness. Some sysadmins therefore mix disks from different vendors and batches.

Hence, I would not want to run a raid with 3TB disks and only use raid-5. During those days, if only another disk crashes you have lost all your data.

Hence, that article is correct, and he is not a scaremonger. Raid-5 is obsolete if you use large drives, such as 2TB or 3TB disks. You should instead use raid-6 (two disks can fail). That is the conclusion of the article: use raid-6 with large disks, forget raid-5. This is true, and not scaremongery.


In fact, ZFS has therefore something called raidz3 - which means that three disks can fail without problems. To the OT: no raid-5 is not safe. Neither is raid-6, because neither of them can not always repair nor detect corrupted data. There are cases when they dont even notice that you got corrupted bits. See my other thread for more information about this. That is the reason people are switching to ZFS - which always CAN detect and repair those corrupted bits. I suggest, sell your hardware raid card, and use ZFS which requires no hardware. ZFS just uses JBOD.

Here are research papers on raid-5, raid-6 and ZFS and corruption:
http://hardforum.com/showpost.php?p=1036404173&postcount=73
 
brutalizer said:
The trend is clear. In the future when we have 10TB drives, they will not be much faster than today. This means, to repair an raid with 3TB disks today, will take several days, maybe even one week. With 10TB drives, it will take several weeks, maybe a month.
Click to expand...

While I agree with the general claim that the larger HDDs (1.5, 2, 3TBs) are best used in RAID 6, your claim about rebuild times is way off.

I think it is not unreasonable to assume that the 10TB drives will be able to read and write at 200 MB/s or more. We already have 2TB drives with 150MB/s sequential speeds, so 200 MB/s is actually a conservative estimate.

10e12/200e6 = 50000 secs = 13.9 hours. Even if there is 100% overhead (half the throughput), that is less than 28 hours to do the rebuild. It is a long time, but it is no where near a month! Try to back your claims in reality.

And you have again made the false claim that "ZFS - which always CAN detect and repair those corrupted bits". ZFS can usually detect corrupted bits, and can usually correct them if you have duplication or parity, but nothing can always detect and repair. ZFS is safer than many alternatives, but nothing is perfectly safe. Corruption can and has happened with ZFS, and it will happen again.
 
Here is my view on the question.

I personally own a 5x2TB RAID 5 array and I believe it to be a safe option.

What I do to avoid problems in the future is scrub my array every month. This process takes about 13 hours to scrub through my 10TB so it's not too bad. What I mean by scrub is by executing this command via CLI.

"echo repair > /sys/block/$array/md/sync_action"

What this does is force the system to go through and check the consistency of the array and if it finds any, fixes the inconsistent stripe from parity.

The positive side effect of this repair scan this is that if I encounter a URE while reading form every sector of the drives, my array can mark the sector as bad and map it to a re-allocated sector and correct it using the parity that exists because the array isn't degraded.

This way, when I do lose a drive, the chances of a brand new URE manifesting itself in my array during the rebuild since the last repair check is much lower than if i haven't done anything and say my array is a year or 2 old.


Also, if a URE is encountered during a rebuild, it should not be a catastrophic array failure. If it is then you should really get a better RAID controller. If you encounter a URE during a rebuild, you should only lose that single stripe in the worst case and that would likely just result in a single corrupt file.

Furthermore, given the nature of your usage in storing video, you would likely be able to recover the file with a corrupt stripe and play it anyways. You may just seen a few weird pixels (or a skipped frame or 2) for a moment in the video as it plays through the corrupted section. Heck, I've even played a movie that was only 50% downloaded in bit-torrent and watched as the player skipped through the video to play only the parts that were downloaded, but the parts that were complete played perfectly. But of course this kind of corruption is much more catastrophic for other file types (especially compressed archives) in which case you would lose the whole file.

You can always mount your degraded array in read-only mode and read all your data off. The only catastrophic failure to a RAID 5 array is if you completely lose 2 drives where you cannot read any data off either anymore. This case is indeed getting more and more likely as drives increase in size but not in reliability, but I do not think that with 5x2TB drives this is a real concern yet unless you have very highly important data.
 
SirMaster said:
Also, if a URE is encountered during a rebuild, it should not be a catastrophic array failure. If it is then you should really get a better RAID controller. If you encounter a URE during a rebuild, you should only lose that single stripe in the worst case and that would likely just result in a single corrupt file.
Click to expand...

That is not the worst case. If you are unlucky, the corrupt stripe could be in your filesystem metadata, and you could lose a lot more than one file.

But I agree with you that it is a good practice to regularly scrub the array. mdadm works as you explained, and many hardware RAID cards also have the ability to do automatic scrubbing periodically (I have done it with Areca and 3Ware cards).

Still, it is best to use RAID 6 with HDDs larger than 1TB. Or in the case of a video collection, something like UnRAID or FlexRAID makes a lot of sense.
 
john4200 said:
That is not the worst case. If you are unlucky, the corrupt stripe could be in your filesystem metadata, and you could lose a lot more than one file.
.
Click to expand...

Oh, good call. I hadn't really thought of that situation. Yeah, it would most definitely be a pretty catastrophic failure.
 
instead of constant real time back up (which is good to have), you can always add a snapshot backup that is stored on an external drive or network for peace of mind.
 
john4200 said:
While I agree with the general claim that the larger HDDs (1.5, 2, 3TBs) are best used in RAID 6, your claim about rebuild times is way off.

I think it is not unreasonable to assume that the 10TB drives will be able to read and write at 200 MB/s or more. We already have 2TB drives with 150MB/s sequential speeds, so 200 MB/s is actually a conservative estimate.

10e12/200e6 = 50000 secs = 13.9 hours. Even if there is 100% overhead (half the throughput), that is less than 28 hours to do the rebuild. It is a long time, but it is no where near a month! Try to back your claims in reality.

And you have again made the false claim that "ZFS - which always CAN detect and repair those corrupted bits". ZFS can usually detect corrupted bits, and can usually correct them if you have duplication or parity, but nothing can always detect and repair. ZFS is safer than many alternatives, but nothing is perfectly safe. Corruption can and has happened with ZFS, and it will happen again.
Click to expand...
This is getting tiresome.

Look, if you have a raid in production, and you try to repair it, you will never reach full speed. And, when you repair a raid, what is important is random read and random write. Have you seen benchmarks for random read and random write? A 2TB disk, capable of 150MB/sec sequential workload, is down to maybe 0.4MB/sec when there is random workload. In worst case: 0.4MB/sec random workload during repair, and in addition, the raid is in production and in heavy use - well it is going to take a loooong time to repair the raid. In worst case, the users will write more than 0.4MB/sec to the raid, which is more than the speed which repairs the drive - the repair will never even finish! With 10TB disks, a month is not unrealistic. Remember, this comes from Adam Leventhal, developer of ZFS - which knows quite much about Enterprise storage halls. You have apparently no clue what happens in such halls.

So before you accuse others of lying or what not, I suggest you do some research first; is your thougths reasonable? No? Then maybe you should not post. If you dont know if your thoughts are reasonable, study some more.

And regarding ZFS providing data integrity - Ive told you several times there are research papers about the safety of ZFS. I told you to read the research papers - which you refuse to do. Then why do you post things about ZFS if you dont know jack?

As I said, you are getting tiresome.
 
I think everyone is getting tired of you preaching ZFS actually. Everyone seems to tout it like it's the second coming...but it isn't. There's a reason few enterprise environments use it and stick with RAID instead.
 
Blue Fox said:
I think everyone is getting tired of you preaching ZFS actually. Everyone seems to tout it like it's the second coming...but it isn't. There's a reason few enterprise environments use it and stick with RAID instead.
Click to expand...
There is a reason everyone talks about ZFS. It is not hype, there is a strong reason to use it.

I see you did not read the links from CERN I posted. In one of the links, CERN says that data corruption is found in ALL solutions, even Enterprise solutions. Some Enterprise solutions even use ext3 or something like that - with some additional software. And ext3 is not safe, as research shows. Nor is hardware raid 5 nor 6 - as research shows.

I suggest you read the research papers instead, so you understand the greatness of ZFS. Otherwise, if you dont read them, you will not understand what is the point with ZFS. But maybe you do not value your data.
 
I mentioned nothing of it's advantages or shortcomings, merely the way in which you and others tout it. People do not want to be preached to or have things shoved down their throat. Me reading some research paper has no relevance on what I had posted. I know exactly what the advantages and disadvantages of ZFS are. It helps to stay informed.

If ZFS was so ungodly spectacular, it would sell itself and would not need you or others preaching and pushing it like a used car salesman or religious zealot. You can't dispute the fact that it is not used in most enterprise environments and if you don't know why, maybe you should research into that instead of sticking to your one research paper like it's the bloody Bible.
 
Blue Fox said:
I mentioned nothing of it's advantages or shortcomings, merely the way in which you and others tout it. People do not want to be preached to or have things shoved down their throat. Me reading some research paper has no relevance on what I had posted. I know exactly what the advantages and disadvantages of ZFS are. It helps to stay informed.
Click to expand...
I doubt you know the advantages and disadvantages of ZFS. Tell us them, I suspect more people wants to know.


Blue Fox said:
If ZFS was so ungodly spectacular, it would sell itself and would not need you or others preaching and pushing it like a used car salesman or religious zealot. You can't dispute the fact that it is not used in most enterprise environments and if you don't know why, maybe you should research into that instead of sticking to your one research paper like it's the bloody Bible.
Click to expand...
Of course it is not used in most environments - but that is not a valid argument.

You are saying something like: "Windows is the most common OS, that must mean that Windows is better than Linux, better than Solaris, better than Mac OS X, etc - better than everything. There is a reason everyone use Windows. You can not dispute this fact. If you really think that Windows is inferior, maybe you should research into that instead of talking about how bad Windows is"

Hardly a valid argument, is it?

As CERN says, lots of (all?) Enterprise storage solutions are not safe with respect to data integrity. Why do you think CERN is migrating to ZFS?
 
brutalizer said:
I doubt you know the advantages and disadvantages of ZFS. Tell us them, I suspect more people wants to know.
Click to expand...
Thanks, but the last thing I'm doing is preaching about something. The merits of ZFS are not the point here anyway. The preaching part is.
brutalizer said:
Of course it is not used in most environments - but that is not a valid argument.

You are saying something like: "Windows is the most common OS, that must mean that Windows is better than Linux, better than Solaris, better than Mac OS X, etc - better than everything. There is a reason everyone use Windows. You can not dispute this fact. If you really think that Windows is inferior, maybe you should research into that instead of talking about how bad Windows is"

Hardly a valid argument, is it?

As CERN says, lots of (all?) Enterprise storage solutions are not safe with respect to data integrity. Why do you think CERN is migrating to ZFS?
Click to expand...
The reason why Windows is the most commonly used OS is the same reason why ZFS is seldom used in enterprise environment: support. As a result of this, it is and isn't a good fit for the majority of aforementioned environments respectively.
 
Blue Fox said:
Thanks, but the last thing I'm doing is preaching about something. The merits of ZFS are not the point here anyway. The preaching part is.
Click to expand...
Sure. I talk to people and they say they know everything about ZFS. Then it turns out, they dont know that hw raid and ZFs is a very bad combination. Nor do they know anything about data integrity. I understand you did not catch up on latest research on data corrupion. If you did, you would also migrate to ZFS. In other words, no, you dont know jack about ZFS. You do believe hw raid is good enough. But as I said, you most probably dont value your data. But, please, dont say you know about ZFS because you obviously dont.


Blue Fox said:
The reason why Windows is the most commonly used OS is the same reason why ZFS is seldom used in enterprise environment: support. As a result of this, it is and isn't a good fit for the majority of aforementioned environments respectively.
Click to expand...
Do you claim that Windows is most used becuase Windows have the best support? Have I understood you correct? And do you claim that ZFS does not have good support?

Que???
 
You must log in or register to reply here.
Back
Top