Two drives failed on RAID 5 array - need advice

[MountainX]

I have an Areca ARC-1120 (Firmware Version V1.39 2005-12-13) with 8 drives set up as two arrays (raid sets) of 4 drives each. One array uses four WDC WD4000YR-01PLB0 drives in a RAID 5 setup. The second array uses four Maxtor drives. The Maxtor drives have been failing regularly. Initially all four Maxtor drives were the same model. Now there are two MAXTOR STM3320620AS and two Maxtor 7L300S0 drives set up as RAID 5.

Tonight both Maxtor 7L300S0 drives failed and the entire array therefore failed. What advice can anyone give me about attempting to recover the data. The drives were not making any strange noises. The failure happened during a lot of disk activity, and that has been the pattern in the past too. (In the past only a single drive has failed at any time.) I have noticed that when a large amount of data is written to the RAID array, a drive tends to fail (or be identified as having failed). The last two times this happened I was backing up my laptop to the RAID array. I mention this because it makes me think the drives didn't physically fail.

What steps can I take to try to recover from this?


Once I recover, then I will need to figure out why the Maxtor drives are failing so regularly. The drives are not from the same lot. Seagate tech support told me there is no firmware problem related to the drives that have failed. The drives are not hot -- they run at 26 to 28 degrees C. In the last two years I have replaced about 6 or 7 Maxtor drives in this raid set, but none of the WD's.

[devman]

Does seatools say the drives are bad? You might be able to image the disk to a new one and reinsert it in to the array. This is coming from my admittedly low understanding of building RAID5 arrays but I figure I'd toss that one out there.

In the future I'd recommend a backup solution to avoid things like this!

[Rev Lemmon]

I cant help you with recovering the data, search the formus for Raid 5 Recovery there were some excellent software suggestions that may help you out. Reguardless of of how well they were running if the maxtors are failing that often I wouldnt take anymore chances with them and just replace them all. You could also try taking the drives out and putting them into another DO NOT FORMAT THEM!!! Just see if bios and windows disk manager can recognize them. If they can then they may not have physically failed, the raid card may have just dropped them from the arrary for some arbirtary reason. Depending on the size of the drives it may just be easier to replace them all with larger drives and run a raid 1 setup instead, another option for you to consider, hope that help you abit. Good Luck.

[MountainX]

I cant help you with recovering the data, search the formus for Raid 5 Recovery there were some excellent software suggestions that may help you out. .

I'm not finding any good posts after searching this forum for raid 5 recovery. If anyone has a link, please post it. Thanks.

[MountainX]

Does seatools say the drives are bad?
In the future I'd recommend a backup solution to avoid things like this!

It will take me a good bit of effort to test with seatools. I'll probably do that tomorrow.

What backup solutions do you recommend for 2 TB of data?

[MountainX]

Depending on the size of the drives it may just be easier to replace them all with larger drives and run a raid 1 setup instead, another option for you to consider, hope that help you abit. Good Luck.

I would like to move from raid 5 to raid 10 using larger drives. And then maybe I'll use another computer with JBOD as the backup...

[devman]

It will take me a good bit of effort to test with seatools. I'll probably do that tomorrow.

What backup solutions do you recommend for 2 TB of data?

Only backup what is important to you. If the whole 2TB is important to you, WD makes some 2TB external drives.

Part of budgeting storage space is ensuring that you have enough space reserved for backing up neccissary/important/irreplaceable items.

That being said, you likely didn't come here to be told you should backup your data. Do the seatools thing, hopefully the drives are not bad.

[satterth]

If the drives function enough that you can IMAGE them to a file then do so. After which you can load the 3 or 4 images files into RAID Reconstructor (http://www.runtime.org/raid.htm) and it can probe the images and attempt to destripe the array into a single image to which you can attempt to mount and read your data back or even go further into recovery depending if the file system is corrupt or not.

If the drives are damaged in other ways I'm sure the many companies like this http://www.werecoverdata.com/ would be more than willing to help you for a cost.

[MountainX]

If the drives function enough that you can IMAGE them to a file then do so.

Imaging sounds like a good idea. I'll be surprised if the drives are so bad that I can't image them... we'll see tomorrow.

So far, I pulled 1 of the two failed drives out and replaced it with a new drive. (I wish I had not done this.) The raid array was visible in Windows Explorer and I could see all the folders some files on it (but not all of the files were there). Then I shut down the computer.

My plan is to proceed something like this:
1. boot the file server up with the original drives - including the two that were marked as failed. (I'll pull out the new replacement drive I stuck in). I want to see if things look any better with this simple step.
2. I have heard that deleting and recreating the raid set with the Areca controller (but not deleting the data) can be a useful step. I will research this further before trying it.
3. I will test each failed drive with seatools.
4. I will image all 4 drives, if possible. I have to figure out where to store the images...
5. I'll try Runtime's recovery tools -- I've used their drive image XML software before, and I trust them.
6. Once I recover the data, I'll replace all 4 Maxtor drives with 4 larger, more reliable drives and configure it as RAID 10. (I'll have to figure out how to restore the recovered data -- particularly if I place the images on the new drives that I'll be putting into service...
7. I'll update the firmware on the raid controller
8. I will have to come up with a backup plan - either a JBOD NAS device, or, if I can afford it, Amazon S3.

Am I leaving out any major steps?

[Rev Lemmon]

Sounds like a good plan stan.

[gjvrieze]

Imaging sounds like a good idea. I'll be surprised if the drives are so bad that I can't image them... we'll see tomorrow.

So far, I pulled 1 of the two failed drives out and replaced it with a new drive. (I wish I had not done this.) The raid array was visible in Windows Explorer and I could see all the folders some files on it (but not all of the files were there). Then I shut down the computer.

My plan is to proceed something like this:
1. boot the file server up with the original drives - including the two that were marked as failed. (I'll pull out the new replacement drive I stuck in). I want to see if things look any better with this simple step.
2. I have heard that deleting and recreating the raid set with the Areca controller (but not deleting the data) can be a useful step. I will research this further before trying it.
3. I will test each failed drive with seatools.
4. I will image all 4 drives, if possible. I have to figure out where to store the images...
5. I'll try Runtime's recovery tools -- I've used their drive image XML software before, and I trust them.
6. Once I recover the data, I'll replace all 4 Maxtor drives with 4 larger, more reliable drives and configure it as RAID 10. (I'll have to figure out how to restore the recovered data -- particularly if I place the images on the new drives that I'll be putting into service...
7. I'll update the firmware on the raid controller
8. I will have to come up with a backup plan - either a JBOD NAS device, or, if I can afford it, Amazon S3.

Am I leaving out any major steps?

If I were in your boat (which totally sucks, I feel for ya!!) I would grab Spin Rite, and take each drive to a spare machine, and run the recovery on each drive alone, if is able to complete the operation, even with "unrecoverable" sectors, run Ghost or the like on the drives, and make an image, then put the image on the new drives, and plug them into the controller, and hope for the best.. If Spin Rite, fails to make a complete pass, you running out of easy luck, but usually if the drive is not making noises, Spin Rite, can get back 100% of the drive....

I have had customers, that had Windows BSODing every bootup, with missing file errors, run Spin Rite, Level 2, and in a few hours, the machine boots, and can make 100% perfect images to go to the new drive.... I cannot stress how highly, I recomend this tool, and if you do not believe, I did not either, till I started trying it, in a little over a year, I believe I recovered about 50-100 drives, and pleased tons of customers... It is best money EVER spent on data!!!! I am just a huge Steve Gibson fan, and in no way benefit from saying this.... There are going to be a few people who do not believe and even slam me for posting this, but I guaranty, usually, non of them have even tried, they just think it is BS from the specs, but you have to try it!!!!

[devman]

If I were in your boat (which totally sucks, I feel for ya!!) I would grab Spin Rite, and take each drive to a spare machine, and run the recovery on each drive alone, if is able to complete the operation, even with "unrecoverable" sectors, run Ghost or the like on the drives, and make an image, then put the image on the new drives, and plug them into the controller, and hope for the best.. If Spin Rite, fails to make a complete pass, you running out of easy luck, but usually if the drive is not making noises, Spin Rite, can get back 100% of the drive....

I have had customers, that had Windows BSODing every bootup, with missing file errors, run Spin Rite, Level 2, and in a few hours, the machine boots, and can make 100% perfect images to go to the new drive.... I cannot stress how highly, I recomend this tool, and if you do not believe, I did not either, till I started trying it, in a little over a year, I believe I recovered about 50-100 drives, and pleased tons of customers... It is best money EVER spent on data!!!! I am just a huge Steve Gibson fan, and in no way benefit from saying this.... There are going to be a few people who do not believe and even slam me for posting this, but I guaranty, usually, non of them have even tried, they just think it is BS from the specs, but you have to try it!!!!

I've actually witnessed spin rite in action, it works well as far as I can tell.

(I also come from a tech repair shop background)

[gjvrieze]

I've actually witnessed spin rite in action, it works well as far as I can tell.

(I also come from a tech repair shop background)

Thanks for the thumbs up, trust me, the more you see it in action, the more you become in aw!!!!

[Seated]

For someone who is more versed in the issues with using non-raid hard drives, could that be the issue here? If you reboot your computer, does the raid controller still see those 2 drives as being bad? If so, can you go into the RAID program and reactivate those drives and see if they work?

I'm just wondering if this is a raid/non-raid hard drive issue.

[MountainX]

First of all, thanks for all the great replies. I am so glad I found this forum!

Here is what I can report. I let the server sit turned off for a day. As I said I would, I put the original drives back in the raid set that had failed and booted the computer. Before Windows loaded I entered the raid bios. The 8-channel Areca 1120 was set up with two volume sets, each with 4 drives. The volume set with the failed drives is completely missing now. Previously, it was present but marked as "failed". All four drives are recognized and all 4 are listed as "free".

Both of the drives that were previously marked as failed have the following info:
Media Errors: 0
SMART Read Error Rate: N.A.
SMART spinup time 252(63)
SMART reallocation count 253(63)
SMART seek error rate 253(0)
SMART spinup retries 252(157)
SMART calibration retries 252(223)

I'm not sure what any of those numbers mean. The non-failed drives have different numbers, but they are also both a different model drive.

Here are the same values for one of the other (non-failed) drives:
0
100(6)
96(0)
100(36)
79(30)
100(97)
N.A.

I do have a license for Spin Rite, so I will go ahead and use that as a next step. Do you think it would be OK to hook the drives to a spare computer using a SATA-to-USB adapter?

Thanks again for all the great feedback.

[gjvrieze]

First of all, thanks for all the great replies. I am so glad I found this forum!

Here is what I can report. I let the server sit turned off for a day. As I said I would, I put the original drives back in the raid set that had failed and booted the computer. Before Windows loaded I entered the raid bios. The 8-channel Areca 1120 was set up with two volume sets, each with 4 drives. The volume set with the failed drives is completely missing now. Previously, it was present but marked as "failed". All four drives are recognized and all 4 are listed as "free".

Both of the drives that were previously marked as failed have the following info:
Media Errors: 0
SMART Read Error Rate: N.A.
SMART spinup time 252(63)
SMART reallocation count 253(63)
SMART seek error rate 253(0)
SMART spinup retries 252(157)
SMART calibration retries 252(223)

I'm not sure what any of those numbers mean. The non-failed drives have different numbers, but they are also both a different model drive.

Here are the same values for one of the other (non-failed) drives:
0
100(6)
96(0)
100(36)
79(30)
100(97)
N.A.

I do have a license for Spin Rite, so I will go ahead and use that as a next step. Do you think it would be OK to hook the drives to a spare computer using a SATA-to-USB adapter?

Thanks again for all the great feedback.

It is ok hooking the drives up via USB for a lot of things, but Steve says that Spin Rite cannot get real low access to the drive through USB and recommends direct connections whenever possible....

I bet if you run those drives throw Spin Rite and then boot up, I bet your array will come back online, and get your data OFF those drives!!!

[MountainX]

Here's an update on what I have done. As I said previously, I believe my HDDs are physically good. At this point, my controller is not finding my raidset at all -- probably because after the two drives failed, I replaced one of them with a new spare and then (without doing any rebuild, recovery, etc.) I reverted back to the original disk that I felt still had all my data on it. So my next task was to get the controller to recognize the raidset. Here are my steps:

1. started computer with only the good drives in the raidset
2. entered raid bios
3. raidset was not found.
4. turned off computer
5. started computer with no drives in the raidset
6. entered raid bios
7. inserted the two good drives
8. raid set was found. Here is the info:
-member disks:4
-raid state: incomplete
-total capacity: 1200.4GB
-free capacity: 1200.4GB
-min disk size: 300.1GB
-member disk channels: 5-7x
9. inserted the last failed disk (channel 6)
10. channel 6 is listed as free and channels 5 & 7 are listed as members
11. inserted the first failed disk (channel 8)
12. all 4 disks are now listed as free.
13. removed all disks while still in raid bios
14. inserted just the two good disks
15. raid set was found and channels 5 & 7 are listed as members

I am trying to decide what to do next. Should I insert the last failed disk (channel 6) and rescue the raid set using the Areca's "rescue raidset" function?

To complicate matters, remember that I'm using all the original disks. Immediately after the problem (before I was thinking clear) I replaced channel 8 with a new spare disk. I did not attempt any rebuild, rescue or recovery of any type. I simply checked the RAID bios, booted the computer, open Windows Explorer and looked around, then turned the computer off. Then I reverted to using all 4 of the original disks and I set this new spare disk aside. I still have it available if I need to use it.

In looking at this, I find it interested that the other raidset in the same computer and attached to the same controlller has not had any drives fail in the last two years while a lot of the Maxtor Maxline III 300GB drives have been failing - more than half a dozen (maybe a dozen of them). They were failing so often at one point that I had to go to my local computer store and purchase more spares. I could only get the DiamondMax 21 320GB drives. I purchased two of those. Neither has failed while the Maxtor Maxline III 300GB drives keep failing like crazy.

[[LYL]Homer]

Didn't read everything, but bad cables?

[MountainX]

I don't think the cables are the issue, although I don't know for sure. The other raidset with 4 WD drives is attached to the controller with the same brand cables installed at the same time and none of those drives are failing. Furthermore, in this raidset only the Maxline III drives are failing. The DiamondMax drives, which use the same cables, are not failing.

[MountainX]

I updated my controller firmware and restarted the server. Now I get a strange situation where the 3 inserted drives (see earlier post for details) that should be part of one raidset are seen as two different raidsets. Here are the details. Note that channel 6, which was the last drive that failed, shows good SMART stats.

After firmware update:

Raid Set Hierarchy
Raid Set IDE Channels Volume Set(Ch/Id/Lun) Volume State Capacity
Primary Raid Set Ch01 00v64kArc1120 (0/0/0) Normal 1200.0GB
Ch02
Ch03
Ch04
Second Raid Set Ch05
Missing
Ch07
Failed
Second Raid Set Missing
Ch06
Missing
Failed

IDE Channels
Channel Usage Capacity Model
Ch01 Primary Raid Set 400.1GB WDC WD4000YR-01PLB0
Ch02 Primary Raid Set 400.1GB WDC WD4000YR-01PLB0
Ch03 Primary Raid Set 400.1GB WDC WD4000YR-01PLB0
Ch04 Primary Raid Set 400.1GB WDC WD4000YR-01PLB0
Ch05 Second Raid Set 320.1GB MAXTOR STM3320620AS
Ch06 Second Raid Set 300.1GB Maxtor 7L300S0
Ch07 Second Raid Set 320.1GB MAXTOR STM3320620AS
Ch08 N.A. N.A. N.A.

IDE Drive Information

IDE Channel 5
Model Name MAXTOR STM3320620AS
Serial Number 5QF1KEGX
Firmware Rev. 3.AAE
Disk Capacity 320.1GB
Current SATA Mode SATA150+NCQ(Depth32)
Supported SATA Mode SATA150+NCQ(Depth32)
Device State NORMAL
Timeout Count 0
Media Error Count 0
SMART Read Error Rate 108(6)
SMART Spinup Time 96(0)
SMART Reallocation Count 100(36)
SMART Seek Error Rate 79(30)
SMART Spinup Retries 100(97)
SMART Calibration Retries N.A.(N.A.)

The SMART Attribute(Threshold) Is A Normalized Value, The Value Is The Larger The Better.
If The Attribute Value Is Smaller Than The Threshold Value, The Disk Is In Unstable State

IDE Drive Information

IDE Channel 6
Model Name Maxtor 7L300S0
Serial Number L62YH1XG
Firmware Rev. BANC1G10
Disk Capacity 300.1GB
Current SATA Mode SATA150+NCQ(Depth32)
Supported SATA Mode SATA150+NCQ(Depth32)
Device State NORMAL
Timeout Count 0
Media Error Count 0
SMART Read Error Rate N.A.(N.A.)
SMART Spinup Time 252(63)
SMART Reallocation Count 253(63)
SMART Seek Error Rate 253(0)
SMART Spinup Retries 252(157)
SMART Calibration Retries 252(223)

The SMART Attribute(Threshold) Is A Normalized Value, The Value Is The Larger The Better.
If The Attribute Value Is Smaller Than The Threshold Value, The Disk Is In Unstable State.

IDE Drive Information

IDE Channel 7
Model Name MAXTOR STM3320620AS
Serial Number 5QF17CLZ
Firmware Rev. 3.AAE
Disk Capacity 320.1GB
Current SATA Mode SATA150+NCQ(Depth32)
Supported SATA Mode SATA150+NCQ(Depth32)
Device State NORMAL
Timeout Count 0
Media Error Count 0
SMART Read Error Rate 108(6)
SMART Spinup Time 96(0)
SMART Reallocation Count 100(36)
SMART Seek Error Rate 79(30)
SMART Spinup Retries 100(97)
SMART Calibration Retries N.A.(N.A.)

The SMART Attribute(Threshold) Is A Normalized Value, The Value Is The Larger The Better.
If The Attribute Value Is Smaller Than The Threshold Value, The Disk Is In Unstable State

[MountainX]

Both of the drives that were previously marked as failed have the following info:
Media Errors: 0
SMART Read Error Rate: N.A.
SMART spinup time 252(63)
SMART reallocation count 253(63)
SMART seek error rate 253(0)
SMART spinup retries 252(157)
SMART calibration retries 252(223)

I'm not sure what any of those numbers mean. The non-failed drives have different numbers, but they are also both a different model drive.


FYI, I found this info:
The SMART Attribute(Threshold) Is A Normalized Value, The Value Is The Larger The Better.
If The Attribute Value Is Smaller Than The Threshold Value, The Disk Is In Unstable State.

That indicates these drives are probably OK, as I suspected.

[gjvrieze]

Ah, that sucks if it is a controllers issue, if you want, you could either hook the drives up to another Areca card, (either a friends controller, or someone here on [h]) or you could order another Areca card just to test, and return after you know for sure, it sucks, but those man well be your only ways to go at this point....

[MountainX]

I think the problem is that the Maxtor Maxline III drives (and other Maxtor models) are not very compatible with many RAID controllers -- that's what I'm told. I think the drives are in good working condition and I think the controller is good. I just think (based on what tech support is telling me) that the Maxtor drives flake out under heavy load due to some drive bios limitations.

I think these drives will be OK to use in non-RAID situations. And I think this controller will be fine with other drives in the second raidset. (The WD drives work great in the 1st raidset.)

Now, I just have to figure out how to get the raidset and volume set recognized again without losing my data... my options are:
1. use the controller's "rescue raid" function
2. use the controller's function to create a raidset without initializing (so data isn't lost)
3. some other option I haven't thought of yet... anyone have ideas?

[gjvrieze]

I would first thing: disable NCQ and even run the drives in SATA 150 if necessary....
Then I would go with "1. use the controller's "rescue raid" function"
Set "HDD Read Ahead Cache"=Disable Maxtor
Set "Disk Write Cache Mode"=Disabled
And see what happens....

And please keep posting your results!!

If you had too, you could even make a backup image of each drive before messing with them under the raid card, if the data is fairly important, it make worth the time, and you may even be able to back them up onto compatible drives, and "mount" them easily under the Areca.... I know Seagate's 7200.10s work great on the 1280ML, and they are cheap, so even if you had to grab 4 of them, the data is prolly worth way more then the cost of the drives:-)

[MountainX]

Thanks gjvrieze. I appreciate your advice. I'll keep posting my results.

At the moment I am cloning each of the 4 HDDs from the raidset.

I'm using the following command to clone them:
dd if=/dev/sda of=/dev/sdb conv=notrunc,noerror
from http://www.linuxquestions.org/questions/linux-newbie-8/learn-the-dd-command-362506/

Before I could use that I had to find out how to tell which device was which HDD because the source and target are the same model drive and same firmware etc. I used this:
udevinfo --query=all --path $(udevinfo -q path -n /dev/sda)
that way I could make sure /dev/sda was really the drive I wanted as my source in the dd command.

Once I have all 4 cloned, I'll do exactly what gjvrieze suggested.

Then I'll back up all the data and get rid of the Maxtor Maxline III drives. I have ordered 4 WD RE2 drives.

(I probably won't be able to post again until Saturday)

[MountainX]

I just received a response from Areca tech support. Here it is:

it looks like the second failed drive didn't been failed correctly.
the suggest procedure will be :
1. power off system and put only 5 and 7 drive in
2. check the raidset status, if it is incompleted, please activated it.
after you activated it, the raidset status should be failed, and the volume is failed too.
3. put the drive 6 in, after it recognized, check the raidset and volume status.
if i am correct, the raidset and volume status will become degraded, you can boot into your system to check the data inside.
4. if the data in volume is correct, you can put the new hotspare drive8 in to rebuild the raidset.

[gjvrieze]

I just received a response from Areca tech support. Here it is:

it looks like the second failed drive didn't been failed correctly.
the suggest procedure will be :
1. power off system and put only 5 and 7 drive in
2. check the raidset status, if it is incompleted, please activated it.
after you activated it, the raidset status should be failed, and the volume is failed too.
3. put the drive 6 in, after it recognized, check the raidset and volume status.
if i am correct, the raidset and volume status will become degraded, you can boot into your system to check the data inside.
4. if the data in volume is correct, you can put the new hotspare drive8 in to rebuild the raidset.

Very cool, I hope this works out for ya, and the damn controller/drive "bug" did not "wipe" your data......:)

[MountainX]

I worked! My data appears to be there (I checked several files and folders). I inserted the new spare as in the last step and the array is rebuilding now :)

I have not yet changed the cache settings in the controller as gjvrieze suggested. However, I will do that as soon as the rebuild completes. I figured I couldn't/shouldn't change those settings before solving the raidset problems.

My plan is to let the rebuild finish, then change the cache settings to work around the Maxtor issues, then back up all the data, then change the Maxtor drives for my new WD RE2 drives when they get here next week. (I'll use the Maxtor drives in a non-raid setting because I have 7 of them and they all show good SMART stats, etc.)

I'll update again if anything changes, but I think this is the resolution of this issue now.

[LordBritish]

OMG RAID sounds so complicated.

I've never used RAID in my life. I heard so many stories of people who have lost data because of it (or because they didn't understand it).

[MountainX]

I ONLY use raid for my important data and I will continue to use raid. But I'll also do better at backing up.

[gjvrieze]

I worked! My data appears to be there (I checked several files and folders). I inserted the new spare as in the last step and the array is rebuilding now :)

I have not yet changed the cache settings in the controller as gjvrieze suggested. However, I will do that as soon as the rebuild completes. I figured I couldn't/shouldn't change those settings before solving the raidset problems.

My plan is to let the rebuild finish, then change the cache settings to work around the Maxtor issues, then back up all the data, then change the Maxtor drives for my new WD RE2 drives when they get here next week. (I'll use the Maxtor drives in a non-raid setting because I have 7 of them and they all show good SMART stats, etc.)

I'll update again if anything changes, but I think this is the resolution of this issue now.

Glad to hear that, the Areca controllers are worth the money if you value your data!!
If you change the cache settings or drive speeds/NCQ while it is online, the settings do not get applied till after the system reboots next....

[MountainX]

I am pretty impressed by the Areca controller's robustness in this situation. The rebuild is completed now and all the data is present.

I changed all the controller's config settings as suggested by gjvrieze in an earlier message.

I am now backing up all the data. (I'm just copying the files to a TeraStation. I still need a good backup solution, particularly because I use hardlinks (http://forge.novell.com/modules/xfmod/project/?ntfslink).)

Overall, I'm very, very happy with the outcome and I learned a few things in the process. Thanks for all the help.

[Pocatello]

glad to see that you got your data back!

I'm thinking of getting the 8-port Areca Hardware raid and stumbled on this post.

[MountainX]

Everything turned out fine. The Areca controller and all HDDs are still humming along great. :)