Limit chances of RAID data corruption

J

Jeroen1000

Limp Gawd
Joined
Sep 17, 2010
Messages
266
Hey everyone,

I did my best to google around and it has just gotten me a tad afraid. Data can get corrupted just about everywhere if you just read long enough. But, in real live, what causes of corruption can go undetected? And if this applies, can remain uncorrected forever?

I've made it this far:

1. Without ECC RAM, data can get corrupted in RAM. The OS, nor RAID controller can do anything about this, correct?
2. What can be done against the RAID-5 write hole. I've found some hints implying a battery backup unit is to be used? I thought that was only needed if you cached writes or something?
3. Some software of system crash corrupts data. Not much can be done here I suppose...
4. Faulty hard drive controller corrupts data?

I know a backup would solve all of this. Too bad backing up everything I have is not doable. Minimizing a potential corruption event is the next best thing for me.
And if data gets silently corrupted between backups, the backup will also be corrupted.

Cheers,
Jeroen
 
Last edited:
The best solution is to keep good backups it is unlikely the same file will get corrupted on both. Although it could be corrupted unnoticed.

breakdown

1. just use ECC ram :)
2. AFAIK ZFS raid-z solves this problem
3. AFAIK copy-on-write should solve this (ie. ZFS).
4. Routine checksummming?

I'm not an expert so take my response with caution.
 
I've shelled out for an Areca raid controller doubting between this and ZFS for some time. Won't go into the 'why' but ZFS did not tick all the boxes.

1. Could do this. Doubt my mobo supports ECC RAM, so another board may be required.
2. No can do
3. Same as 2
4. I'll need to google this.


Excluding RAM as a potential cause, we still have the hard drive that can corrupt stuff.
And I'd also like to know how likely the RAID-controller is to corrupt data. I'd wish that a full backup were possible but that would get too expensive. No such thing a an affordable tape drive AFAIK.
 
Jeroen1000 said:
Excluding RAM as a potential cause, we still have the hard drive that can corrupt stuff.
And I'd also like to know how likely the RAID-controller is to corrupt data. I'd wish that a full backup were possible but that would get too expensive. No such thing a an affordable tape drive AFAIK.
Click to expand...

Well it's not like a RAID controller corrupts data all the time but it does happen enough to where you should have a seperate backup of that data. Also a backup doesn't have to be a tape drive. It can also be multiple external hard drives or another file server/NAS.
 
To backup 24 2 TiB drives, I'd need an identical array. Unless I've missed something big, is this the "cheapest" way to backup such a large amount of data?

Also, ZFS combats silent data corruption on a file system level. Isn't there a technology that hardware RAID can use that does the same?
It seems the option most DIY users prefer is just to live with it:D
 
HW raid only has parity, if there is a parity error, there is no way for the array to know is it is due to an error in a data block or in a parity block.

ZFS has an extra hashing check to determine the correct value.

(that is if I'm not mistaking ;) )
 
Why do say no go on a ZFS setup?
If you are seriously concerned about data integrity there is nothing better than a ZFS setup...especailly for the amount of storage your talking about. With that many 2TB drives, even a totally random bit flip is a real possibility over time.
 
I am assuming your aversion to ZFS is because performance is paramount? They are all correct about ZFS being the most reliable as far as data integrity goes...
 
Hardware RAID is just the way to go with Windows as a requirement. But I do agree the ZFS filesystem might have been the better choice for bulk data storage. There is no way back now though as I'll be using an Areca card + an HP sas-expander.

I'll be raiding 2 * 12 drives and opting for raid 6 with both raid sets. That should be enough to cover for hardware faillure.

This brought me to my 'plan' to counter random bit flips/ silent data corruption as much as possible. Is there really nothing more that can be done apart from using quality hardware:

  • A-brand PSU
  • Sufficient cooling
  • UPS unit
  • ECC RAM?

Even with the above measures there is still (a significant if I combine the sources I found) threat of an:

  • URE
  • Silent data corruption due to firmware error, hardware error (ECC algorithms in hard drives or flaky hardware in general), bent cables, cosmic rays :)rolleyes:), ...

If there really is not anything else, I hope reports of silent data corruption are greatly exaggerated:)

@oldskool. For my video collection performance is not really an issue. But for some other goals it will probably become an issue. Which goes without saying I'm not looking forward to rerip my collection if disaster strikes.
 
I am essentially in the same situation, although my primary reason for going hardware RAID was that I am OCD about performance (even if I don't necessarily need it), and because it is in a Windows machine (although it has since been virtualized). But I would say your fears of silent data corruption are vastly exaggerated. Just use good hardware, backup power, and have the card run checks on the array and you should be OK.
 
Jeroen1000 if you haven't actually gotten the hardware yet I'd reconsider. If you need to do windows no problem. I'd urge you to consider Gea's all-in one concept as a solution. For example:
- Scrap the monster Areca and go 3x Intel SASUC8I or similar.
- Take some of the leftover money towards a better quad core CPU and more ram.
- A small boot disk (2.5" drives?) or two in a tiny hardware raid 1 carrier.
- ESXi on the boot drive with a small datastore for at least the ZFS VM (Nexenta, OI, ect.). You could put your windows VM on that drive OR take a small amount of your big ZFS pool and create a datastore (NFS) for your windows VM.
- Pass through the SATA cards to the sotrage VM and create you large pool however you like using ZFS. SHare it it out to the Windows machine via CIFS and map drive letters as you see fit.
- Remember, if you use OI you can try out VMXNET3 driver on both VM's and and have a 10GBe internal connection between the two VM's through the ESXi virtual switch. (I have done this. A little more to setup but it works!)


Given the extra money towards a little better CPU and memory you will find performance is not going to be an issue. Most especially if it all is being accessed over a single gigabit line.

All the above may sound complicated but it really is not bad. A few months ago I knew nothing at all about ZFS, solaris, and almost nothing about VMware. Since then I have learned a ton and successfully setup and tested rigs with everything I described above. It really works! If your willing to put some time into playing with this and testing you may find you can get ALL the ZFS benefits and plenty of performance while also being able to run a windows machine all from the same physical server. There are of course plenty of people here willing to help out to get you up and running.

PS- You would need to go with a Xeon and server board that had at least 3 PCIe x8 slots and VT-D support. Since you were thinking of ECC ram this might be a good idea anyway.
 
Last edited:
Silenus, I have read your post with great interest as ZFS has captured my interest in the past.
What I felt back then, still applies now: hardware raid does look a lot simpler and will require less exploring on my part to get it up and running.

Back then, I have promised myself to experiment with ZFS first before trusting any imporant data to it. I still feel ZFS is a bit unproven in comparison with hardware raid and since my time to devote to it is rather limited, I just went the easy and proven way.

Some hardware accelerated ZFS would be very cool, actually:). I bet it can be implemented on dedicaded hardware just as it is being done with "regular" raid. To cut it short, I will eventually look into ZFS but for the time being I'm putting my marbles in the hardware raid basket. If I do it the proper way I'll hopefully be fine.


edit: it seems the industry is working on a joint initiative for data integrity: http://www.lsi.com/storage_home/partners/alliance_partners/data_integrity_initiative/
 
Last edited:
Fair point. Traditional hardware raid is simpler overall. I have a lot of experience with hardware raid setups where I work and well...it's been my my experience that for anything beyond a simple RAID 1, unless your using (expensive) raid edition drives (TLER) than large arrays are not terribly stable and very easy to disturb. It's funny because when I was searching for a NAS/backup solution for the company I work for I specifically did NOT want a large hardware RAID setup (because of the disk expense). :)

I guess I figure that if a solaris based ZFS system is good enough for some of the biggest and fastest enterprise databases on the planet...it's good enough for me. :) In the end we built a simple server with a supermicro board and Xeon quad core with 8GB ram. I put 10 Samsung F4 2TB in it, RAIDz2 with a hot spare (extra cautious on disk failure). No L2ARC or ZIL as they are not needed. This mostly holds company wide backups (including critical business data) and is a general purpose NAS for temporary data that gets moved around. This thing gets hammered on every day and has been running continuously without the slightest hiccup for months now. It auto scrub's once a week and hasn't shown a single error yet. I feel comfortable with it at this point.

Having said that...I did spend a lot time experimenting, practicing, and testing setups with ZFS well before I deployed this amchine. So as you say spending time and experimenting is important before feeling comfortable with something. I hope in the future you do experiment some more with ZFS. It's good times. Best of luck on your big build sir!
 
I dont know what you need for ZFS to be proven, but its not just used by consumers here, its in use across a wide array of companies with crucial data concerns.
 
I'll be using Hitachi drives drives and those should be fine without TLER or equivalents (assuming you are talking about them dropping out due to time outs). Mechanically wise, it is generally believed they are of good quality.

Reading your experiences with it, I'm quite impressed about ZFS working so well for you. Just like you, I feel you need to really experiment with new stuff and at the very least test some faillure scenario's. I'm eager to experiment but then my build plans will get even further behind lol.

BTW I have quite a build thread typed up already. If the thing goes south I'll be venting my anger there:))
 
With RAID 6 will the array not be able to fix an error, it has the data 3 times so if 1 out of the 3 is different then the other 2, on a check it would replace that with one of the others? Or what?
 
RAID 6 has extra parity, but it still doesn't guarantee correct data by just taking the majority of equal results. However small the chance may be that two drives generate a concurrent error it's not impossible. And if an error occurs during a rebuild of a disk, you're back down to RAID5 guesswork.

Of course it's also possible for a file to have errors, and still match a crc/hash check. Chances of that happening are ridiculously low, for an Y bit crc you would need EXACTLY (x *) 2^Y bit errors to have the same crc. (correct me if I'm wrong)

But crc is only an error detection system, which needs extra parity info to be able to reconstruct the data (something like RAID, but also things like .par files) RAID has the problem of being on a hardware level, while CRC is based on the file level.
ZFS combines both technologies in software (a file-system-aware hardware device would be very complicated)

(did that make sense? It's still early for me.)
 
You must log in or register to reply here.
Back
Top