Using RAID-5 Means The Sky Is Falling

[drescherjm]

Do you at least know whether it is higher or lower than 1 error in 10^7 ?

I hope you realize that an error rate of 10^-7 is crazy high.

I agree. That 1 error in every 10^7 is absolutely ridiculous.

 

[brutalizer]

I don't think that the exact error rate matters, but the point is, that error rates are far higher than the disk URE of 10^16 or so. As reported by cern and Amazon. There are so much that can go wrong, sata cables, etc.

And I don't think they have faulty setups, I believe their setups are fine, probably much better than anyone here can plan. But, it might happen they have faulty hardware. For instance, it was only when one company switched to zfs, they detected a faulty fibre channel switch, that they missed. The earlier storage did not detect any data corruption. Zfs was the first to notice. Faulty hardware can happen anyone, but is their design good? It is another question.

 

[JoeComp]

I don't think that the exact error rate matters...

Well, there's your problem.

Maybe I'll get a job in the finance industry. If I lose a few million dollars, I'll tell my boss, "I don't think the exact dollar amount matters."

 

[brutalizer]

I dont think it is a problem, because it is just statistics that may be wrong. We only estimate the true error rate and we have no way of finding it. It would be bad to believe these numbers are exact. We always have an confidence interval, not a point estimation. Besides, it is impossible to pin point the very exact number, you only have an estimate. So nat many people dealing with real life scenarios, believes a scenario of 45.6% probability is better than 45.1%. Mostly we try to draw conclusions when evidence is strong. And 45.6% vs 45.1% are not strong evidence for any outcome. However, 10^7 vs 10^16 is strong evidence. It does not really matter if it is 10^7 or 10^8 when we compare to 10^16. So that might be your problem, if you literally believe your disk, every 10^16 bits produces an error. These numbers are not exact.

In finance, we dont focus to much on the exact numbers. We just want to see how strong the evidence is, is it big difference or small difference? If there are 53% you win money, then dont buy (the exact probability might be 49%). You only buy when you are convinced, such as 60-65% or so.

 

[Zarathustra[H]]

Do you at least know whether it is higher or lower than 1 error in 10^7 ?

I hope you realize that an error rate of 10^-7 is crazy high.

If I had to wager a guess, I would guess that it is currently not as bad as 10^-7, but it is very difficult to know exactly.

Also, my pool consists of all reasonably new drives, who knows what might happen as they deteriorate.

The CERN study is a snapshot over a large number of systems in use of varying ages. The assumption has to be that as they age, they become worse.

If I had to wager I guess most errors occur at some point in the writing process, as disks are reasonably stable once written. A smaller amount corrupt over time once committed to disk, as magnetic charge is lost, etc.

So, should our denominator be our writes or our reads? Reads is certainly easier to measure, but reads might be more significant.

Also, to my knowledge ZFS doesn't log and report every parity error it fixes. It heals them and goes on.

Some of these errors may happen during a scrub, at which point I get a report of how much data was repaired, but it is reported as the total data stored in a given block. The mismatch in that block could be caused by a single bit, so using the block total would highly overrepresent the error rate. At least I would know about them in this scenario.

Other repairs happen on the fly every day, as data is accessed, and parity is checked, and if incorrect fixed, and no report is flagged, unless there is a URE, in which case a read error is logged against the hard drive it came from.

There may be a better way to log silent error repairs, but if there is, I am not aware of it. I have done a lot with the ZFS command line tools, but have not looked into detailed error logging yet.

All I know from observing my scrubs is that it is not uncommon at all.

 

[Zarathustra[H]]

80% of all errors they saw was from a incompatible 3ware / WD storage combination. Although the remaining 20% is still a high number.

Agreed.

IMHO the incompatibility bug IS relevant to be included given the fact that there are countelss combinations of controllers, drives, cables, motherboards, etc. out there and if you want to understand the error rates of typical deployed systems, well, typical deployed systems ARE going to have some of these issues.

That being said, even if those 80% are eliminated, this brings us to the 10^-8 level, which is still pretty damned bad.

 

[JoeComp]

Clearly there are people with faulty setups and faulty hardware with crazy high error rates.

But that does not change the fact that the silent error rates for HDDs are far lower than the URE rates for HDDs. Nor does it change the fact that it is possible to configure and maintain systems with silent error rates far, far below the numbers reported in the CERN study, which is really a useless report given the crazy high error rates.

Anyone with errors rates anywhere near that high needs to be fixing their setups rather than worrying about how to detect and correct errors.

 

[Zarathustra[H]]

Anyone with errors rates anywhere near that high needs to be fixing their setups rather than worrying about how to detect and correct errors.

The philosophy here is that these crazy error rates can strike anyone at any time. Controller and drive firmware is a vast wasteland of bugs, and even the most well read IT professional can wind up with something that doesn't play nice together and results in silent corruption, and when this silent corruption occurs, because it is silent, it goes unnoticed.

Better safe than sorry, in my book.

 

[JoeComp]

even the most well read IT professional can wind up with something that doesn't play nice together and results in silent corruption, and when this silent corruption occurs, because it is silent, it goes unnoticed.

If it is anywhere near 10^-7 rate, then any competent "IT professional" should have easily detected it during configuration, testing, or burn-in, or production testing. It is not difficult to detect "silent" errors if the rate is that high. They are only really silent if you do not run any tests that can detect them.

It seems some people here do not understand how crazy high 10^-7 is.

It is like driving a car that is about to fall apart into multiple pieces and worrying about wearing your seat belt.

 

[Zarathustra[H]]

It seems some people here do not understand how crazy high 10^-7 is.

Well yes, it is very surprisingly ridiculously high. It is one bit of error per every 1.19MB, which to your point is rather ridiculous.

But if it is even 0.1% of that rate, it is a real problem.

 

[Zarathustra[H]]

Well yes, it is very surprisingly ridiculously high. It is one bit of error per every 1.19MB, which to your point is rather ridiculous.

But if it is even 0.1% of that rate, it is a real problem.

Then again, we don't know exactly how the number was arrived at.

Maybe they meant one in ever 10^7 blocks has silent data corruption. As little as one bit error could corrupt any given block.

If you use 128KB blocks, that could mean one bad block in 1.19TB, which is still bad, but a little bit more believeable. It results in the bit error rate (if the assumption is made that every bad block has just one corrupted bit) at the 10^13 level.

 

[brutalizer]

If it is anywhere near 10^-7 rate, then any competent "IT professional" should have easily detected it during configuration, testing, or burn-in, or production testing. It is not difficult to detect "silent" errors if the rate is that high. They are only really silent if you do not run any tests that can detect them.

Again, it is really really really hard to detect silent corruption. Maybe that is the problem, as your view of storage does not align with the big server enterprise storage vendors.

Frequently we see threads here about people asking why zfs reports errors, and in the end it turns out to be faulty psu, sata cables, dim, etc. Often, these people have used another filesystem with no problems. I even read a thread here, we're someone rejected zfs because he got error reports, whereas his old linux solution never reported errors. His conclusion was that zfs was "too sensitive" and fragile, as it saw errors all the time. I suggested that maybe there were problems with his setup, which he flat out rejected. He went back to linux and ext3 or xfs, to stop all data corruption reports. On Linux he never saw error reports, whereas zfs reported them frequently. Why? Is zfs problematic, or was his logic flawed?

I told you about the company using fibre channel switches without problems, until they switched to zfs, and immediately saw data corruption reports. They did not know one switch was faulty (which they had used for a long time) until zfs reported errors. Had you asked them before zfs, they would have said their storage was 100% safe. I find it hard to fathom that anyone reading all threads here where zfs reporting cables were loose, or all stories about zfs detecting this or that problem - still do believe silent errors don't exist, or is easy to detect. Especially when you consider these stories showed no corruption before switching to zfs.

The actual error rates in real life is far higher than 10^16, which both CERN and Amazon reports in large scale studies, which means these studies are reliable and statistically significant. So you can put some amount of faith in them. Who would trust a study only on 1000 disks??? No, you need to examine millions of disks, as cern and Amazon did. For instance, NetApp (enterprise storage vendor with deep knowledge) examines 1.5 million disks in their expensive servers at customers, and see very often errors that their raid card did not detect. These are called silent errors, for a reason, and therefore hard to detect. This is why you need zfs, and you should scrap hw raid. Sun realized this more than a decade ago, and even today, people don't understand:

-I talked to my friends, and in total we have 100 disks, and we have never seen any data corruption reports from Windows. Besides, it is easy to detect silent errors. I don't know why it is called "silent" but probably such errors are easy to detect. So, CERN, Amazon and NetApp clearly knows less about storage than me and my friends, because they see errors all the time, whereas we do not. I reject their studies, and have never read any study on this. What is more probable you think? I know more than these large companies, or they know more but I believe I know this stuff but actually I don't. You draw your conclusions.

(Question, what do zfs say on your storage? Maybe you have problems which only zfs detects?)

 

[drescherjm]

Again, it is really really really hard to detect silent corruption.

At a 10^-7 rate it is extremely easy to detect "silent" corruption. If you can even call it silent. I mean every few MB of data written would be corrupt.

 

[JoeComp]

At a 10^-7 rate it is extremely easy to detect "silent" corruption. If you can even call it silent. I mean every few MB of data written would be corrupt.

Right. And "silent" errors are easy to detect when setting up a system. One easy way to detect the problem is with badblocks, especially if the rate is higher than the URE rate.

Even on a production system it is extremely easy to detect these types of errors when the rates are high. All you need to do is write a file with a known pattern and/or checksum and then read it back and compare.

It is not some mystical problem. Any competent computer user can test and detect it when the rates are high.

 

[Zarathustra[H]]

Right. And "silent" errors are easy to detect when setting up a system. One easy way to detect the problem is with badblocks, especially if the rate is higher than the URE rate.

Even on a production system it is extremely easy to detect these types of errors when the rates are high. All you need to do is write a file with a known pattern and/or checksum and then read it back and compare.

It is not some mystical problem. Any competent computer user can test and detect it when the rates are high.

I agree, if present during setup, they should be caught, unless there's an irresponsible lack of setup testing.

So there is little excuse for the WD firmware controller bug/incompatibility discussed in the CERN study.

That being said, systems degrade over time. This is where most of my concerns would be.

 

[brutalizer]

At a 10^-7 rate it is extremely easy to detect "silent" corruption. If you can even call it silent. I mean every few MB of data written would be corrupt.

I dont think the silent errors are at 10^-7, but all errors together have that rate - on average. The silent errors are only a small part of all errors.

With that said, you will not know you have silent errors in your system, unless you explicitly check for that. For instance, writing a known bit pattern to disk, and read it back and compare, just as JoeComp explained. CERN did this for three weeks and saw that indeed, bit patterns differed on some storage servers. Before this test, CERN thought their storage was safe. It was not. But how many of you do this check? You can setup a daemon doing this constantly in the background so you will get a notification whenever something gets weird. For instance, sata cables gets loose, psu will degrade, ram dimms get faulty, etc. These things happen as time passes. So yes, if your data is important, you maybe should have a process constantly verifying your storage for every read/written block.

 

[drescherjm]

Before this test, CERN thought their storage was safe.

I agree, if present during setup, they should be caught, unless there's an irresponsible lack of setup testing.

I hope whomever came to that conclusion (that the storage was safe) was fired. This is an extreme level of incompetence for not doing any burn in testing.

But how many of you do this check?

I do at least 1 week of testing every new server (3 days minimum testing the ram with the rest testing the disk subsystem). I also do at minimum a 4 pass badblocks read /write test on every single drive that I put in my servers. In this test 8 total passes are made over a disk and if a single bit is out of place I will reject the drive.

 

[brutalizer]

That is great that you do burn in tests, which means you only deploy safe servers. Which means you should not get fired.

But.... have you considered the possibility that CERN do burn in tests, but after a year or so, some components start to fail in the well tested servers? Vibrations might shaken sas cables, etc? There is a famous video where someone shouts aloud at a server, and thanks to Dtrace in Solaris, you can see that the disks gets stressed of all the screaming. This vibration might cause silent corruption. Google this, its a cool video.

Maybe you should constantly do a read/write check at all times, to catch errrors, just as as mentioned? It does not suffice to test the storage only once, at deployment? You should install a daemon in the background that does this constantly. As Amazon explained, they see errors all the time, as time passes, and Amazon deploys only well tested servers. As time passes, hardware will fault, the question is, will you catch all errors? New hardware might not report errors after three days burn in, but might report errors after one week, or month, or...?

Or, you could use zfs, which does this constant testing for you. It seems that zfs catches many types of errors, including when ram dimm gets bad. ECC does catch ram errors too, which is good, but if you dont have ECC ram, then zfs will catch failing ram dimm immediately when zfs notices data corruption. According to comp sci researchers, zfs catches ALL types of errors, not "many types of errors", but all types. To me, the researchers conclusion is huge, and it might explain the success of zfs. But of course, you could develop your own daemon that does this check constantly in the background, hopefully researchers will spend time on it to verify it works as you intended, and is safe.

 

[drescherjm]

That is great that you do burn in tests, which means you only deploy safe servers. Which means you should not get fired.

They should be fired because they were not monitoring the data. They either did no burn in testing at all or after burn in testing they did not monitor their servers at all. This is again on based on the 10^-7 number. If that is the true rate of errors whomever was in charge of the storage was not doing their job at all.

Maybe you should constantly do a read/write check at all times, to catch errrors, just as as mentioned? It does not suffice to test the storage only once, at deployment? You should install a daemon in the background that does this constantly. As Amazon explained, they see errors all the time, as time passes, and Amazon deploys only well tested servers. As time passes, hardware will fault, the question is, will you catch all errors? New hardware might not report errors after three days burn in, but might report errors after one week, or month, or...?

Or, you could use zfs, which does this constant testing for you.

With zfs and mdadm/ btrfs I experience a URE about every 500TB to 1 PB of data read (which obviously nothing like 1 every XXX MB that 10^-7 says). This comes from weekly scrubs of a 70TB dataset for several years + other CRC/ MD5/SHA verification done in my backups.

 

[JoeComp]

With zfs and mdadm/ btrfs I experience a URE about every 500TB to 1 PB of data read (which obviously nothing like 1 every XXX MB that 10^-7 says). This comes from weekly scrubs of a 70TB dataset for several years + other CRC/ MD5/SHA verification done in my backups.

What are you considering a "URE"?

I think common usage from HDD manufacturers is that a URE means a read error that the drive reports as an error.

If you are including "silent" errors in that count, then it is a misleading use of the term URE.

If possible, the best way to report errors would be to report both the URE rate and the "silent" error rate.

 

[drescherjm]

I guess URE was not the best word here. I am talking about the rate of having a single corrupt block. I go several months of weekly scans between seeing a single check sum mismatch on zfs scrubs ...

Also I investigate ever single drive that has reported corruption. If a drive reports a single mismatch I pull it out of my servers and give it a 4 pass badblocks. if it reports a single bad block I rererun the test 2 more times. If it reports a single bad block in that I RMA the drive.

 

[Zarathustra[H]]

I guess URE was not the best word here. I am talking about the rate of having a single corrupt block. I go several months of weekly scans between seeing a single check sum mismatch on zfs scrubs ...

Also I investigate ever single drive that has reported corruption. If a drive reports a single mismatch I pull it out of my servers and give it a 4 pass badblocks. if it reports a single bad block I rererun the test 2 more times. If it reports a single bad block in that I RMA the drive.

So what are you talking about here?

Are you talking about one of those that gets logged in the READ WRITE CKSUM columns on the right when you do a "zpool status"?

I've only ever had those on drives which are imminently failing.

What is a lot more common though - at least on my setup - is seeing something like the below "196K repaired" during a scrub, that never shows up in any logs.

Presumably this is the real time ZFS healing we hear so much about, but details on this are surprisingly limited on teh googlez.

pool: zfshome
state: ONLINE
  scan: scrub in progress since Fri Jan 16 15:48:18 2015
        16.5T scanned out of 43.6T at 585M/s, 13h32m to go
        196K repaired, 37.68% done
config:

    NAME                                            STATE     READ WRITE CKSUM
    zfshome                                        ONLINE       0     0     0
     raidz2-0                                      ONLINE       0     0     0
       gptid/85faf71f-2b00-11e4-bc04-d8d3855ce4bc  ONLINE       0     0     0
       gptid/86d3925a-2b00-11e4-bc04-d8d3855ce4bc  ONLINE       0     0     0
       gptid/87a4d43b-2b00-11e4-bc04-d8d3855ce4bc  ONLINE       0     0     0
       gptid/887d5e7f-2b00-11e4-bc04-d8d3855ce4bc  ONLINE       0     0     0  (repairing)
       gptid/eb518a3c-63d9-11e4-8721-000c29dbe1ad  ONLINE       0     0     0
       gptid/b62d0aa1-638a-11e4-8721-000c29dbe1ad  ONLINE       0     0     0
     raidz2-1                                      ONLINE       0     0     0
       gptid/56fb015b-2bfc-11e4-be49-001517168acc  ONLINE       0     0     0
       gptid/576cde68-2bfc-11e4-be49-001517168acc  ONLINE       0     0     0
       gptid/57dbbac1-2bfc-11e4-be49-001517168acc  ONLINE       0     0     0
       gptid/584a4dcc-2bfc-11e4-be49-001517168acc  ONLINE       0     0     0
       gptid/58f4ec2f-2bfc-11e4-be49-001517168acc  ONLINE       0     0     0
       gptid/abd7d2b7-63cf-11e4-8721-000c29dbe1ad  ONLINE       0     0     0
    logs
     mirror-2                                      ONLINE       0     0     0
       da14p1                                      ONLINE       0     0     0
       da7p1                                       ONLINE       0     0     0
    cache
     gptid/89f2024c-4010-11e4-bf9d-000c29dbe1ad    ONLINE       0     0     0
     gptid/8a137ec5-4010-11e4-bf9d-000c29dbe1ad    ONLINE       0     0     0

errors: No known data errors

 

[drescherjm]

So what are you talking about here?

Are you talking about one of those that gets logged in the READ WRITE CKSUM columns on the right when you do a "zpool status"?

I've only ever had those on drives which are imminently failing.

What is a lot more common though - at least on my setup - is seeing something like the below "196K repaired" during a scrub, that never shows up in any logs.

In the case of zfs I am talking about both of these.

I also monitor the SMART (Current_Pending_Sector_raw,Offline_Uncorrectable_raw,Reallocated_Sector_Ct_raw,Temperature_Celsius_raw and UDMA_CRC_Error_Count_raw) for every single disk using nagios.

 

[pxc]

For some strange reason people seem to think that everything changes when we talk about ________, and that a statistical possibility becomes absolute certainty.

lol, that perfectly explains 90% of the dumbest comments in every news thread.

 

[mikeblas]

I see that Backblaze is using triple-parity (their own implementation) with their latest design.

https://www.backblaze.com/blog/vault-cloud-storage-architecture/

This isn't their own design; it's just an implementation of erasure encoding.

 

[JoeComp]

This isn't their own design; it's just an implementation of erasure encoding.

When I say "their own implementation" I mean custom code and/or custom hardware. That is the usual meaning of the phrase, I believe. And that is what Backblaze did.

You seem to be confused about the meaning of "erasure coding". That is simply a name for a general algorithm (or technique, whatever you want to call it). It is not the name of a specific implementation. Or maybe you are confused about the English language? Given what I wrote (and you quoted), I cannot make any sense of what you wrote in response.

 

[brutalizer]

They should be fired because they were not monitoring the data. They either did no burn in testing at all or after burn in testing they did not monitor their servers at all.

You talk about zfs scrub and monitoring your data regularly - and monitoring your data and servers is what every serious sysadmin should do. (ZFS is only one of the tools to help you monitor your servers for faulty hardware and data integrity). The time before, did you use hardware raid? And if you did, did you monitor your data back then? Or, have you always been aware of data corruption, and the need to constantly monitor your storage to catch failing hardware? There must have been a time, when you were not aware of this problem, and did not monitor your data? When did you become aware of this problem? My point is, maybe you were just faster than other sysadmins? But there was a time, when even you did not know about this? So why blame others, if even you did not know some time ago? (There are many people even today who are not aware - but they are not sysadmins).

It seems that more and more serious sysadmins becomes aware of this problem, and this knowledge has spread to CERN too. Which might be the reason CERN switched to ZFS on their long term tier-2 and tier-3 storage, according some links from CERN out on the internet.

 

[Zarathustra[H]]

You talk about zfs scrub and monitoring your data regularly - and monitoring your data and servers is what every serious sysadmin should do. (ZFS is only one of the tools to help you monitor your servers for faulty hardware and data integrity). The time before, did you use hardware raid? And if you did, did you monitor your data back then? Or, have you always been aware of data corruption, and the need to constantly monitor your storage to catch failing hardware? There must have been a time, when you were not aware of this problem, and did not monitor your data? When did you become aware of this problem? My point is, maybe you were just faster than other sysadmins? But there was a time, when even you did not know about this? So why blame others, if even you did not know some time ago? (There are many people even today who are not aware - but they are not sysadmins).

It seems that more and more serious sysadmins becomes aware of this problem, and this knowledge has spread to CERN too. Which might be the reason CERN switched to ZFS on their long term tier-2 and tier-3 storage, according some links from CERN out on the internet.

I guess my take is monitoring is essential, but often times by the time you start getting error reports in your logs, with traditional hardware RAID it is already too late. You have corrupt data.

With ZFS, there is still time to take action without corrupting any data.

 

[drescherjm]

And if you did, did you monitor your data back then?

Yes, although not as throughly. Back then (1990s) I used crc checks and tested compressed archives ... but certainly not on a weekly bases. I did not have anywhere near 70TB online then. On top of this we always had some form of backup however the backup storage devices were not always reliable.

 

[Zarathustra[H]]

Yes, although not as throughly. Back then (1990s) I used crc checks and tested compressed archives ... but certainly not on a weekly bases. I did not have anywhere near 70TB online then. On top of this we always had some form of backup however the backup storage devices were not always reliable.

Wait...

You don't do regular backups in a production system???

 

[drescherjm]

You don't do regular backups in a production system???

I still do regular backups. This primarily backup for image data (the bulk of our data) is to a dual drive 24 slot LTO2 autochanger and I currently have 200+ tapes. I am budgeting a replacement however due to funding problems (its been significantly harder to get government funding in the last 2 years than it has been any time in the last 20) this will have to wait.

User folders are mirrored from a system with btrfs on top of mdadm raid6 to a server with 2 zfs raidz2 vdevs nightly using rsync. These are also backed up to tape using the autochanger (daily incrementals, weekly differentials, monthly fulls). We have a 1 year retention on the tape and also the daily / weekly / monthly snapshots (on both btrfs and zfs).

 

[brutalizer]

Yes, although not as throughly. Back then (1990s) I used crc checks and tested compressed archives ... but certainly not on a weekly bases

Wow. That is serious. It sounds as you know your job.