Hardware raid rebuild - data corruption

[Bigzaj]

How realistic is it to go through a rebuild of a hardware array and not get data corruption? I went through a rebuild on a Raid 6 array recently, mind you I anticipated high levels of corruption due to some failed Windows indexing and other things...but many video files came back in the right place, full size, full info, etc. but when I played them they were very very corrupt or would not render at all.

Is there always some level of data degradation or corruption that happens with hardware raid in rebuilds?

 

[Neb]

Need more info on your circumstances, you shouldn't be experiencing any corruption with a successful rebuild. I haven't had any degradation or corruption after a successful rebuild on my raids.

 

[drescherjm]

Is there always some level of data degradation or corruption that happens with hardware raid in rebuilds?

If the hardware is working properly you are not supposed to have any corruption at all. Although if you have 2 drives knocked out of a raid6 and during the rebuild any of your disks have a URE you will see corruption on the stripes where a URE occurred. There should not be many of those. Same goes for 1 drive knocked out of a raid 5 with a URE.

 

[houkouonchi]

I upgraded my 20x1TB to 20x2TB by replacing each 1TB drive with a 2TB disk and let it rebuild. Even after 20 rebuilds (to completely replace all the disks) I had no corruption.

 

[Bigzaj]

In this case I knew I would because I had 3-4 failed rebuilds due to a bad SAS Expander, plus windows chkdsk and re-indexing failed twice. I knew there was no hope for the data. I just meant in general with hardware raid vs some other (ZFS).

What do you mean "URE"?

 

[Bigzaj]

I upgraded my 20x1TB to 20x2TB by replacing each 1TB drive with a 2TB disk and let it rebuild. Even after 20 rebuilds (to completely replace all the disks) I had no corruption.

How long did that take?? I have a 12x1.5TB I plan to upgrade to 12x3TB later this year.

 

[houkouonchi]

How long did that take?? I have a 12x1.5TB I plan to upgrade to 12x3TB later this year.

It took about 10 days:

2010-05-25 07:07:10  LINUX VOLUME     Complete Rebuild      009:24:45
2010-05-24 21:45:02  LINUX VOLUME     Start Rebuilding
2010-05-24 21:45:02  MAC VOLUME       Complete Rebuild      000:00:38
2010-05-24 21:44:23  MAC VOLUME       Start Rebuilding
2010-05-24 21:44:23  WINDOWS VOLUME   Complete Rebuild      000:02:33
2010-05-24 21:41:50  WINDOWS VOLUME   Start Rebuilding
2010-05-24 21:41:48  18TB RAID6       Rebuild RaidSet
2010-05-24 21:41:47  IDE Channel #20  Device Inserted
2010-05-24 21:41:03  IDE Channel #20  Device Removed


2010-05-24 21:33:15  LINUX VOLUME     Complete Rebuild      009:18:47
2010-05-24 12:14:27  LINUX VOLUME     Start Rebuilding
2010-05-24 12:14:27  MAC VOLUME       Complete Rebuild      000:00:40
2010-05-24 12:13:47  MAC VOLUME       Start Rebuilding
2010-05-24 12:13:47  WINDOWS VOLUME   Complete Rebuild      000:02:36
2010-05-24 12:11:10  WINDOWS VOLUME   Start Rebuilding
2010-05-24 12:11:08  18TB RAID6       Rebuild RaidSet
2010-05-24 12:11:08  IDE Channel #19  Device Inserted
2010-05-24 12:09:17  IDE Channel #19  Device Removed
2010-05-24 12:09:17  18TB RAID6       RaidSet Degraded


2010-05-24 04:36:36  LINUX VOLUME     Complete Rebuild      009:13:46
2010-05-23 19:22:49  LINUX VOLUME     Start Rebuilding
2010-05-23 19:22:49  MAC VOLUME       Complete Rebuild      000:00:39
2010-05-23 19:22:10  MAC VOLUME       Start Rebuilding
2010-05-23 19:22:10  WINDOWS VOLUME   Complete Rebuild      000:02:34
2010-05-23 19:19:36  WINDOWS VOLUME   Start Rebuilding
2010-05-23 19:19:34  18TB RAID6       Rebuild RaidSet
2010-05-23 19:19:33  IDE Channel #18  Device Inserted
2010-05-23 19:17:37  IDE Channel #18  Device Removed


2010-05-23 17:59:33  LINUX VOLUME     Complete Rebuild      009:27:17
2010-05-23 08:32:15  LINUX VOLUME     Start Rebuilding
2010-05-23 08:32:15  MAC VOLUME       Complete Rebuild      000:00:38
2010-05-23 08:31:37  MAC VOLUME       Start Rebuilding
2010-05-23 08:31:37  WINDOWS VOLUME   Complete Rebuild      000:02:36
2010-05-23 08:29:01  WINDOWS VOLUME   Start Rebuilding
2010-05-23 08:28:59  18TB RAID6       Rebuild RaidSet
2010-05-23 08:28:59  IDE Channel #17  Device Inserted
2010-05-23 08:27:52  IDE Channel #17  Device Removed


2010-05-23 03:39:12  LINUX VOLUME     Complete Rebuild      009:09:30
2010-05-22 18:29:41  LINUX VOLUME     Start Rebuilding
2010-05-22 18:29:41  MAC VOLUME       Complete Rebuild      000:00:37
2010-05-22 18:29:03  MAC VOLUME       Start Rebuilding
2010-05-22 18:29:03  WINDOWS VOLUME   Complete Rebuild      000:02:33
2010-05-22 18:26:29  WINDOWS VOLUME   Start Rebuilding
2010-05-22 18:26:27  18TB RAID6       Rebuild RaidSet
2010-05-22 18:26:27  IDE Channel #16  Device Inserted
2010-05-22 18:23:44  IDE Channel #16  Device Removed


2010-05-22 18:23:34  LINUX VOLUME     Complete Rebuild      009:13:10
2010-05-22 14:39:05  IDE Channel #13  Reading Error
2010-05-22 09:10:23  LINUX VOLUME     Start Rebuilding
2010-05-22 09:10:23  MAC VOLUME       Complete Rebuild      000:01:04
2010-05-22 09:09:19  MAC VOLUME       Start Rebuilding
2010-05-22 09:09:19  WINDOWS VOLUME   Complete Rebuild      000:02:33
2010-05-22 09:06:45  WINDOWS VOLUME   Start Rebuilding
2010-05-22 09:06:43  18TB RAID6       Rebuild RaidSet
2010-05-22 09:06:43  IDE Channel #15  Device Inserted
2010-05-22 09:06:01  IDE Channel #15  Device Removed


2010-05-21 22:15:28  LINUX VOLUME     Complete Rebuild      009:18:18
2010-05-21 12:57:10  LINUX VOLUME     Start Rebuilding
2010-05-21 12:57:10  MAC VOLUME       Complete Rebuild      000:00:40
2010-05-21 12:56:30  MAC VOLUME       Start Rebuilding
2010-05-21 12:56:30  WINDOWS VOLUME   Complete Rebuild      000:02:39
2010-05-21 12:53:51  WINDOWS VOLUME   Start Rebuilding
2010-05-21 12:53:49  18TB RAID6       Rebuild RaidSet
2010-05-21 12:53:49  IDE Channel #13  Device Inserted
2010-05-21 12:53:00  IDE Channel #13  Device Removed


2010-05-21 10:15:21  LINUX VOLUME     Complete Rebuild      009:13:33
2010-05-21 01:01:48  LINUX VOLUME     Start Rebuilding
2010-05-21 01:01:48  MAC VOLUME       Complete Rebuild      000:00:37
2010-05-21 01:01:10  MAC VOLUME       Start Rebuilding
2010-05-21 01:01:10  WINDOWS VOLUME   Complete Rebuild      000:02:35
2010-05-21 00:58:35  WINDOWS VOLUME   Start Rebuilding
2010-05-21 00:58:33  18TB RAID6       Rebuild RaidSet
2010-05-21 00:58:33  IDE Channel #12  Device Inserted
2010-05-21 00:52:47  IDE Channel #12  Device Removed


2010-05-20 21:12:27  LINUX VOLUME     Complete Rebuild      009:17:57
2010-05-20 11:54:29  LINUX VOLUME     Start Rebuilding
2010-05-20 11:54:29  MAC VOLUME       Complete Rebuild      000:00:38
2010-05-20 11:53:51  MAC VOLUME       Start Rebuilding
2010-05-20 11:53:51  WINDOWS VOLUME   Complete Rebuild      000:02:33
2010-05-20 11:51:17  WINDOWS VOLUME   Start Rebuilding
2010-05-20 11:51:15  18TB RAID6       Rebuild RaidSet
2010-05-20 11:51:15  IDE Channel #11  Device Inserted
2010-05-20 11:50:22  IDE Channel #11  Device Removed


2010-05-20 11:49:18  LINUX VOLUME     Complete Rebuild      009:09:57
2010-05-20 02:39:21  LINUX VOLUME     Start Rebuilding
2010-05-20 02:39:21  MAC VOLUME       Complete Rebuild      000:00:39
2010-05-20 02:38:42  MAC VOLUME       Start Rebuilding
2010-05-20 02:38:42  WINDOWS VOLUME   Complete Rebuild      000:02:32
2010-05-20 02:36:10  WINDOWS VOLUME   Start Rebuilding
2010-05-20 02:36:08  18TB RAID6       Rebuild RaidSet
2010-05-20 02:36:07  IDE Channel #10  Device Inserted
2010-05-20 02:33:27  IDE Channel #10  Device Removed



2010-05-20 02:29:06  LINUX VOLUME     Complete Rebuild      009:04:23
2010-05-19 17:24:42  LINUX VOLUME     Start Rebuilding
2010-05-19 17:24:42  MAC VOLUME       Complete Rebuild      000:00:42
2010-05-19 17:24:00  MAC VOLUME       Start Rebuilding
2010-05-19 17:24:00  WINDOWS VOLUME   Complete Rebuild      000:02:44
2010-05-19 17:21:15  WINDOWS VOLUME   Start Rebuilding
2010-05-19 17:21:13  18TB RAID6       Rebuild RaidSet
2010-05-19 17:21:13  IDE Channel #09  Device Inserted
2010-05-19 17:20:24  IDE Channel #09  Device Removed


2010-05-19 17:18:36  LINUX VOLUME     Complete Rebuild      009:20:34
2010-05-19 07:58:01  LINUX VOLUME     Start Rebuilding
2010-05-19 07:58:01  MAC VOLUME       Complete Rebuild      000:00:42
2010-05-19 07:57:19  MAC VOLUME       Start Rebuilding
2010-05-19 07:57:19  WINDOWS VOLUME   Complete Rebuild      000:02:51
2010-05-19 07:54:28  WINDOWS VOLUME   Start Rebuilding
2010-05-19 07:54:26  18TB RAID6       Rebuild RaidSet
2010-05-19 07:54:26  IDE Channel #08  Device Inserted
2010-05-19 07:53:26  IDE Channel #08  Device Removed



2010-05-19 06:13:52  LINUX VOLUME     Complete Rebuild      009:03:06
2010-05-18 21:10:46  LINUX VOLUME     Start Rebuilding
2010-05-18 21:10:46  MAC VOLUME       Complete Rebuild      000:00:38
2010-05-18 21:10:07  MAC VOLUME       Start Rebuilding
2010-05-18 21:10:07  WINDOWS VOLUME   Complete Rebuild      000:02:34
2010-05-18 21:07:33  WINDOWS VOLUME   Start Rebuilding
2010-05-18 21:07:31  18TB RAID6       Rebuild RaidSet
2010-05-18 21:07:31  IDE Channel #07  Device Inserted
2010-05-18 21:06:42  IDE Channel #07  Device Removed



2010-05-18 21:01:25  LINUX VOLUME     Complete Rebuild      009:17:17
2010-05-18 11:44:07  LINUX VOLUME     Start Rebuilding
2010-05-18 11:44:07  MAC VOLUME       Complete Rebuild      000:00:41
2010-05-18 11:43:26  MAC VOLUME       Start Rebuilding
2010-05-18 11:43:26  WINDOWS VOLUME   Complete Rebuild      000:02:38
2010-05-18 11:40:47  WINDOWS VOLUME   Start Rebuilding
2010-05-18 11:40:45  18TB RAID6       Rebuild RaidSet
2010-05-18 11:40:45  IDE Channel #06  Device Inserted
2010-05-18 11:39:55  IDE Channel #06  Device Removed


2010-05-18 07:22:40  LINUX VOLUME     Complete Rebuild      009:22:33
2010-05-17 22:00:06  LINUX VOLUME     Start Rebuilding
2010-05-17 22:00:06  MAC VOLUME       Complete Rebuild      000:00:43
2010-05-17 21:59:23  MAC VOLUME       Start Rebuilding
2010-05-17 21:59:23  WINDOWS VOLUME   Complete Rebuild      000:02:42
2010-05-17 21:56:40  WINDOWS VOLUME   Start Rebuilding
2010-05-17 21:56:38  18TB RAID6       Rebuild RaidSet
2010-05-17 21:56:38  IDE Channel #05  Device Inserted
2010-05-17 21:55:52  IDE Channel #05  Device Removed


2010-05-17 21:55:25  LINUX VOLUME     Complete Rebuild      009:36:53
2010-05-17 12:18:32  LINUX VOLUME     Start Rebuilding
2010-05-17 12:18:32  MAC VOLUME       Complete Rebuild      000:00:41
2010-05-17 12:17:50  MAC VOLUME       Start Rebuilding
2010-05-17 12:17:50  WINDOWS VOLUME   Complete Rebuild      000:02:34
2010-05-17 12:15:16  WINDOWS VOLUME   Start Rebuilding
2010-05-17 12:15:14  18TB RAID6       Rebuild RaidSet
2010-05-17 12:15:14  IDE Channel #04  Device Inserted
2010-05-17 12:14:44  IDE Channel #04  Device Removed

2010-05-17 12:12:50  LINUX VOLUME     Complete Rebuild      009:13:14
2010-05-17 02:59:36  LINUX VOLUME     Start Rebuilding
2010-05-17 02:59:36  MAC VOLUME       Complete Rebuild      000:00:40
2010-05-17 02:58:55  MAC VOLUME       Start Rebuilding
2010-05-17 02:58:55  WINDOWS VOLUME   Complete Rebuild      000:02:47
2010-05-17 02:56:07  WINDOWS VOLUME   Start Rebuilding
2010-05-17 02:56:05  18TB RAID6       Rebuild RaidSet
2010-05-17 02:56:05  IDE Channel #03  Device Inserted
2010-05-17 02:55:31  IDE Channel #03  Device Removed



2010-05-17 02:47:28  LINUX VOLUME     Complete Rebuild      009:14:36
2010-05-16 17:33:00  LINUX VOLUME     Start Rebuilding
2010-05-16 17:33:00  MAC VOLUME       Complete Rebuild      000:00:40
2010-05-16 17:32:19  MAC VOLUME       Start Rebuilding
2010-05-16 17:32:19  WINDOWS VOLUME   Complete Rebuild      000:02:40
2010-05-16 17:29:39  WINDOWS VOLUME   Start Rebuilding
2010-05-16 17:29:37  18TB RAID6       Rebuild RaidSet
2010-05-16 17:29:36  IDE Channel #02  Device Inserted
2010-05-16 17:29:07  IDE Channel #02  Device Removed



2010-05-16 16:55:40  LINUX VOLUME     Complete Rebuild      007:09:41
2010-05-16 09:45:59  LINUX VOLUME     Start Rebuilding
2010-05-16 09:45:41  H/W MONITOR      Raid Powered On
2010-05-16 09:44:18  H/W MONITOR      Raid Powered On
2010-05-16 09:42:30  LINUX VOLUME     Stop Rebuilding       002:47:40
2010-05-16 06:56:24  LINUX VOLUME     Start Rebuilding
2010-05-16 06:56:24  MAC VOLUME       Complete Rebuild      000:00:41
2010-05-16 06:55:42  MAC VOLUME       Start Rebuilding
2010-05-16 06:55:42  WINDOWS VOLUME   Complete Rebuild      000:02:56
2010-05-16 06:52:46  WINDOWS VOLUME   Start Rebuilding
2010-05-16 06:52:44  18TB RAID6       Rebuild RaidSet
2010-05-16 06:52:44  IDE Channel #01  Device Inserted
2010-05-16 06:52:16  IDE Channel #01  Device Removed

 

[drescherjm]

In this case I knew I would because I had 3-4 failed rebuilds due to a bad SAS Expander, plus windows chkdsk and re-indexing failed twice. I knew there was no hope for the data. I just meant in general with hardware raid vs some other (ZFS).

What do you mean "URE"?

Read the first 3 paragraphs of the following:
http://www.datarecovery.net/newsletters/hard-drive-cannot-read.html

 

[brutalizer]

Actually, the risk is high of getting data corruption during rebuild.

The risk of getting unrecoverable error is 1 per 10^16 bits (read the spec sheet on any SAS or Fibre Channel disks). If we talk about SATA disks, then the risk is 1 per 10^15 (read the spec sheets). The risk is very low. But, because you are moving so many bits, the risk is actually quite high of getting an error. You are doing it so many times so the probabilities adds up.

The probability can be calculated by the well known formula
1 - (1 - P)^n
which gives the chance of getting one error, or two errors, or three errors, or four errors, etc.

P is "1 / 10^16"
n is how many bits you move when you repair a raid. Say you have 2TB disks, then n is 2 billion Bytes. Multiply this with 8 to get the number of bits. I will multiply with 10 because it simplifies the calculations.

If you do this, you will get:
1 - (1 - 1/10^16)^(2*10^12) = ???
And I have no calculator that can calculate that probabiliy. But maybe someone else can?

I guess the probability is close to 20% or so.

 

[drescherjm]

Actually, the risk is high of getting data corruption during rebuild.

That is chance of a seeing a URE. In raid6 the raid array will automatically repair the URE if you have only one drive out of the array so the real risk of data corruption is very low. You will have corruption if you have 2 drives out of the array and experience a URE or if you have 1 drive out and hit UREs on two disks in the same stripe. Having two UREs on the same stripe will be highly unlikely to occur unless you have a crazy amount of drives in your array.

I guess the probability is close to 20% or so.

Much lower than that in practice. I have quite a few raid 6 arrays here at work. However this seems to be drive dependent. I mean the Seagate drives I have have a much higher real world URE rate then my hitachi or WDC drives.

 

[bcbgboy13]

What is you hardware configuration - especially is your system ECC capable or not as this may be the cause of your data corruption and not the URE - and I believe the URE on the consumer level HDs is nowadays down to 1 per 10^14 bits ( around 12 TBytes)

Going from your posts you recently went from Asus P8Z68-V to ASUS P8B WS LGA 1155 Intel C206 which coupled with a Xeon is a proper board.

 

[drescherjm]

The cause of this was most likely the bad sas expander.

 

[Bigzaj]

What is you hardware configuration - especially is your system ECC capable or not as this may be the cause of your data corruption and not the URE - and I believe the URE on the consumer level HDs is nowadays down to 1 per 10^14 bits ( around 12 TBytes)

Going from your posts you recently went from Asus P8Z68-V to ASUS P8B WS LGA 1155 Intel C206 which coupled with a Xeon is a proper board.

Yes, that board with Xeon processor and ECC ram is now in my server.

 

[odditory]

The cause of this was most likely the bad sas expander.

Maybe the SAS expander was faulty, maybe his particular motherboard was causing an intermittent incompatibility, maybe it was too hot in the case and the expander chip was overheating which is most often the culprit, but when you are faced with a failed raidset or hardware *appears* to be failing you need to slow down, resist the urge to panic, look at the areca event log and take things one step at a time before you start rebooting windows or letting chkdsk run or throwing rescue raidset keywords at the card, they're very dangerous in many circumstances. Based on the original thread http://hardforum.com/showthread.php?t=1634680 it looks like too many different things were being tried at once before the real problem was identified.

There's a reason Areca doesn't publish all the different rescue raidset keywords, because they only apply case by case, all dependent on specific symptoms. The threads where the keywords are mentioned are dangerous because people tend to be in panic mode just trying anything and will enter them in rapid succession without knowing what each one does or how it applies to their recovery scenario. Example the level2rescue can needlessly wipe out good signatures/metadata in many scenarios.

But to the OP, no data doesn't go corrupt on a RAID6, not without making an effort to corrupt it even if unintentional . Data dies hard. Next time contact Areca support immediately rather than guessing and trying too many things at once. Also, make a habit of keeping an eye on the Areca event log,

 

[brutalizer]

But to the OP, no data doesn't go corrupt on a RAID6, not without making an effort to corrupt it even if unintentional . Data dies hard.

Raid-6 might corrupt data. There is an research paper saying this.

 

[danswartz]

Can you elaborate? I know about the raid5 (6 too) write hole. That only happens in certain specific cases. Are you talking about something else? If so, what?

 

[odditory]

Raid-6 might corrupt data. There is an research paper saying this.

I like ham.