Areca RAID 5 Array - time out error failed array

[tool50]

Hello all,

First of all, I'd like to say any help at all is appreciated. Here's my deal.

I have an ARC-1222 with 8 1TB Seagate Drives (ST31000333AS). Yes, I know Seagate! Unfortunately, that what I have, I realize its less than ideal from reading the posts on here.

Anyway, here's the issue. Its a RAID-5 array and a drive failed (I received an email from the card). I sent in for the 2 day drive warranty advanced replacement). In the meantime, today I received an email from the card saying that a different drive timed out ("time out error") and because of that the array failed.

I'm not in front of the drives right now, but I wanted to get this posted in case others could help. I'll put the log details below. The data on these devices is very important and I have only a partial backup. (The RAID 5 part was the backup).

I'm guessing the drive that "timed out" is just fine or would be fine. I haven't powered the array off, or done any of the various commands as I'm waiting for advice. I still see the volumes in Windows (strangely enough) and I was able to copy a few 1KB files from a folder to a different array, but anything bigger than that wasnt accessible (I'm guessing they were cached somehow?).

Regardless, what are my next steps? I've also emailed ARECA, but since the array and errors are straightforward, I'm guessing someone has some ideas.

Thanks again!
Nate

LOG:
2012-01-14 14:40:36 192.168.001.015 HTTP Log In
2012-01-14 12:54:19 Enc#1 Slot#6 Device Failed
2012-01-14 12:54:19 Raid Set # 000 RaidSet Degraded
2012-01-14 12:54:17 ARC-1222-VOL#000 Volume Failed
2012-01-14 12:53:45 Enc#1 Slot#6 Time Out Error
2012-01-12 03:04:27 H/W Monitor Raid Powered On
2012-01-11 21:25:07 192.168.001.099 HTTP Log In
2012-01-11 21:05:50 192.168.001.099 HTTP Log In
2012-01-11 19:36:33 Enc#1 Slot#5 Device Removed
2012-01-11 19:32:58 Enc#1 Slot#5 Device Failed
2012-01-11 19:32:57 Raid Set # 000 RaidSet Degraded
2012-01-11 19:32:55 ARC-1222-VOL#000 Volume Degraded
2012-01-11 19:32:03 Enc#1 Slot#5 Time Out Error

 

[ABSiNTH]

Check Seagate's as well as Areca's compatibility lists. I had constant RAID5 drop outs with my WD drives, turns out I needed to jumper them in SATA I mode. My problems went away. My point is that there may be some useful information on one of those sheets to help the issue. Hope this helps.

 

[houkouonchi]

An ARC-1222 works just fine with ST31000333AS. They do not with the ST31000340AS but also work good with the ST31000528AS as well.

We have probably 100 machines with ARC-1222 and that model of seagate disk here at work.

The disk was failed (likely due to the timeout) which is what failed the array (not the timeout itself). It does list the volumes as degraded/failed just before the drive actually does timeout.

What does the smart stats of slot 6 look like?

cli64 disk info drv=6

Or clicking the disk in the web interface (the smart values/thresholds)?

To recover rebooting if a newer firmware it should auto revive the failed array. If not you can go in and completely re-create the raid set and volume sets with no init with either the new disk you got or the original slot 5 and then pull the slot after the arrays are created (so its in the correct state) and then put the new slot 5 again in to let it rebuild.

You also might want to not have the machine doing normal taks (leave in BIOS) to try to lesson the load during the rebuild for a greater chance if it completing.

If the array rebuild does not complete then you can mirror slot 6 to a new drive with something like dd_rescue and then try this procedure again.

If you do go as far as re-creating the array with no init be sure to first take a screenshot of the web-interface on the raid set info page to insure that the logical and physical order of the disks is correct.

For example here is mine:



Under the 'devices' column where it lists your raid set hierarchy it will always list these in logical order. When logical order and physical order is correct then the devices should be counting in order from top to bottom. If the physical order is different from the logical order then it will count weirdly.

Here is an example of one where the drive order was changed so physical/logical no longer agree:



This usually only happens if ports were changed or you are using a hotspare.

In this case it was caued by the SFF-8087 cables getting swapped on the controller thus swapping slots 1-4 with 5-8. The physical and logical order must match when re-creating the array from scratch with no init as it is always setup that way when you create the array.

 

[tool50]

All,

Thanks for your help. I eventually decided to power off the system since that appeared to be the next step as you mentioned. When I powered it back up, luckily disk 6 was ok!!! and the array went back to a degraded state. I was able to access the volume and all was OK!. I'm so excited!!!! I've begun copying off additional information which is critical at this time. Thanks everyone for your help. I might be moving to RAID 6 after this..

 

[Trepidati0n]

All,

Thanks for your help. I eventually decided to power off the system since that appeared to be the next step as you mentioned. When I powered it back up, luckily disk 6 was ok!!! and the array went back to a degraded state. I was able to access the volume and all was OK!. I'm so excited!!!! I've begun copying off additional information which is critical at this time. Thanks everyone for your help. I might be moving to RAID 6 after this and also have an external backup of my data so I don't have to make posts like this again.

FTFY

 

[tool50]

Funny, because I had thought that was exactly what this forum was for, helping others. I believe I documented the information fairly well, provided details on what happened before and during the issue, didn't use caps or scream and was patient for responses. Not quite sure what else I could have done to make the post to your liking...

 

[Trepidati0n]

Funny, because I had thought that was exactly what this forum was for, helping others. I believe I documented the information fairly well, provided details on what happened before and during the issue, didn't use caps or scream and was patient for responses. Not quite sure what else I could have done to make the post to your liking...

I wasn't being snarky. I was making a point. Helping others is fine. If we don't repeatedly pound into people that "RAID is not a backup", the problem is never solved.

People with larger arrays are finally starting to come around that RAID6 is now a minimum (the chance of URE with 8 drives @ 1TB each is not trivial during a rebuild). However, a lot of people are still not coming around to actually having a backup of their data.

I just had a HDD fail last night on my WHS server. Unfortunately it was the drive which had most of the backup database. Fortunately I had an external backup of it made just a week ago. I learned long ago that things will break and things I least want to break..DO.

I am glad you didn't make a panic post. I'm glad you went through it calmly. I am glad that I don't get to read a post of "I lost all my data". I'm never glad though to hear that somebody is willing to spend so much money on their data they spent years collecting but not back it up.

 

[tool50]

Yes, now understanding where you are coming from, I completely agree. Here's the deal, I do have an offsite backup (I pay for Crashplan), but since its several terabytes, you can guess it takes a long time to backup to the "cloud" even though I have a decent high speed cable connection. I had 1TB of the 3TB+ in the cloud and I made sure it was the most important 1TB of it first. That said, I'm going out right now to purchase small 3TB NAS type of disk as a backup. I typically HATE these types of drives, because it always seems they fail, but maybe it could be around when I need it, which hopefully isn't often. This primary array hasn't failed me for a good few years now. Thanks again for everyone's help; it is appreciated.