Raid 6 + 2 failed drives + read error on one drive = :(

R

rubicon72

n00b
Joined
Aug 4, 2009
Messages
4
well just as the title says, I have a server with an areca 1280ml and 14 x 1.5 tb seagate hard drives. I recently suffered the simultaneous loss of 2 drives followed by one hard drive giving read errors.

As a result, rebuilding the array is impossible.

Could anyone give me any suggestions to recover the array?

Should I contact a data recovery company to fix that hard drive and proceed with a rebuild?
 
3ware has an ignoreecc command line option for rebuilds, but I don't see anything similar for Areca.

If you have a spare drive and know your way around linux you could try copying the drive with dd using the option "conv=noerror,sync" and using the copy for the rebuild.
 
keenan beat me to it. It's a migraine day, so I'm just getting up here.

However, as soon as you get it rebuilt, you should immediately split the array into two smaller arrays and two hotspares. With 14 disks, you should go with 2x 4+2 and 2 hotspares. Yes, you will lose some capacity, but you also won't explode like this in a two disk failure.
 
AreEss said:
keenan beat me to it. It's a migraine day, so I'm just getting up here.

However, as soon as you get it rebuilt, you should immediately split the array into two smaller arrays and two hotspares. With 14 disks, you should go with 2x 4+2 and 2 hotspares. Yes, you will lose some capacity, but you also won't explode like this in a two disk failure.
Click to expand...

I'm curious if it's really the disks which spewed this error and not rather a incompatibility or a controller glitch. Losing 3 drives on a 14 drive array is like winning the lotto, unless if he lost one and not noticed it until it's too late.
 
Ockie said:
I'm curious if it's really the disks which spewed this error and not rather a incompatibility or a controller glitch. Losing 3 drives on a 14 drive array is like winning the lotto, unless if he lost one and not noticed it until it's too late.
Click to expand...

I've seen a bad batch of drives drop 27 of 48 disks in a san before, so yes, multidrive failures CAN happen.
 
BuBBa said:
I've seen a bad batch of drives drop 27 of 48 disks in a san before, so yes, multidrive failures CAN happen.
Click to expand...

Yes, and I've been in two near plane crashes before (brace for impact and say your prayers), but this is a rarity (just demonstrating that anything CAN happen, but is not likely).

Hence why I made the lotto comment. I would not discourage these findings based on drive failures alone, a bulk failure rate like that could mean many possible scenarios that should be looked at.
 
BuBBa said:
I've seen a bad batch of drives drop 27 of 48 disks in a san before, so yes, multidrive failures CAN happen.
Click to expand...

There used to be paranoia about this in always getting drives from different lots to minimize this risk. Not sure how high that paranoia is these days.
 
Trepidati0n said:
There used to be paranoia about this in always getting drives from different lots to minimize this risk. Not sure how high that paranoia is these days.
Click to expand...

It depends on how well the storage administrator has done his homework. A good administrator would do extensive testing and research before pushing large batches like this into production and a good recovery strategy is implemented.
 
Ockie said:
It depends on how well the storage administrator has done his homework. A good administrator would do extensive testing and research before pushing large batches like this into production and a good recovery strategy is implemented.
Click to expand...

I believe the client got the SAN fully populated from the vendor.
 
BuBBa said:
I believe the client got the SAN fully populated from the vendor.
Click to expand...

That's how we get ours, but we still run tests for sometimes up to a month before we start thinking about production on it. A vendor will have compatible hardware (or at least should), but many things can happen to the drives, especially if your shipper is crazy or you have a bad batch like you mentioned.
 
Is this the program you were talking about keenan? (http://www.garloff.de/kurt/linux/ddrescue/)

Im not a linux user. Should I find a free version of it, install it on a computer, connected the drive with the read error and a free drive to the computer, install the program and run it?

Any help and step by step instruction for something like this would be greatly appreciated
 
Ockie said:
Yes, and I've been in two near plane crashes before (brace for impact and say your prayers), but this is a rarity (just demonstrating that anything CAN happen, but is not likely).

Hence why I made the lotto comment. I would not discourage these findings based on drive failures alone, a bulk failure rate like that could mean many possible scenarios that should be looked at.
Click to expand...

Actually, this is completely wrong, thanks to Seagate 7200.11's.

The firmware bug in these drives was present in the OEM'd "Enterprise" version in a nastier version, where you had somewhere around a 30% chance every power cycle or reset of losing the drive permanently. I personally witnessed a 64 drive system drop somewhere around 20 drives on a power cycle, after having 8 drives DOA. No amount of testing would have caught it, simply because people don't test power cycling or resetting the controllers. Those aren't normal operating conditions.
 
They fixed it on consumer drives, but alot of people probably didn't update. I have one of them and never updated it, don't use raid and have no problems with it. Just have the ole porn collection on it, figure if it goes i have a new reason to go porn collecting again.
 
rubicon72 said:
I thought seagate fixed that firmware problem
Click to expand...

Pfft. They found out about it more than 6 months after drives were in the field, in the thousands upon thousands. Then they told everyone to STFU and wait another 5 months for the fix. Which only PARTLY works, because it only fixes ONE specific bug, doesn't fix it 100%, and doesn't address any other number of OTHER bugs.

The firmware bugs are STILL there. Only Seagate claims otherwise, and they are flat out lying. The 7200.11 is the most unsafe drive out there.
 
You must log in or register to reply here.
Back
Top