ARECA 1220 RAID 6 data recovery help

[pchiang]

Hello:

Wanted to see if anybody might have clues on next steps on recovering data on a ARECA 1220 RAID 6 raidset/volume? The situation is as follows:
I have a RAID6 setup with 8 disks of 2TB each, giving me 12TB usable.
Probably about 5-6 months ago, one of the disks started going in/out of the raidset, triggering rebuilds every 1.5-2 months. Being the (sadly) lazy person that I was, I did not pull out that drive, so rebuilds happened maybe a total of 4-5 times. No biggie I figured.

Just last night, the same same rebuild happened, but this time, unaware to me, the raid set configuration + volume config all disappeared from the areca raid card. The event log just showed a rebuild getting started about 5-6 hours ago, but when I tried to access the volume, it hung. So I ended up rebooting to get to the console and saw that all of the raidset info was gone.

My best guess is that the raidset rebuild did not complete and something happened (maybe that bad disk went bad again??) that caused the raidset config to disappear from the card.

Anyways, I ended up shutting off the box and starting up, and the disk that was bad showed up again.

At this point, I did a few google searches and saw that I could recreate the raidset without a re-init. So I promptly tried that, using a few config variables that I thought I had used previously (64bit LBA vs. 4K blocks, and 64K stripe size vs. 128K). Unfortunately, in all of those cases, a volume check reported a ton of errors.

So wanted to get opinions on next steps? Thinking that I may be able to:
1) Take out the flaky drive, and see if I can force the raid card to be in a degraded RAID6 nodes with 7 disks instead of the expected 8. (??)
2) Leave that drive in and see if we can continue the rebuild?

No clue how to force either of the above tho.

Probably I also need to first find out which settings I had my original raidset set up as - 64bit LBA or 4K blocks, and the strip size). Is there a tool that can help identify that for me?

Thanks in advance for any help,
Peter

 

[mwroobel]

First, post us a screenshot of the RAID manager so we can see what you have now. Second, please post the areca log from the card. Third, Always ask here (or Areca support) before just imputing commands you found on the net to the areca manager, as you can damage things further or completely screw any chance of recovery by doing things wrong. You have a 1220, what is the firmware revision of the card? What brand and model are the drives you have connected? Are they directly connected to do you have an expander in the middle? What operating system are you running?

Next, what exactly HAVE you done so far? RESCUE, SIGNAT, LeVeL2ReScUe ? What order did you do them in (the log will show some of this). Did you just accept the default specs when you initially created the array or did you change them at all? Are the drives still in the same physical connection order they were? Not knowing the specs you created the array with complicates things but doesn't necessarily make this a funeral. As to how to find the info, R-Studio may or may not be able to help.
The most important thing here, do you have a backup of the data? If not, do you have 8 additional available 2TB drives so you can use to make images of the drives currently in your array (so you can't make things worse by trying to repair it yourself and can work with copies of your data instead of the original data) How important is this data (eg is this just your movie and/or MP3 collection or is your business running on this box?)

 

[pchiang]

Thanks mwroobel for the quick response.

Am attaching images for the raidset config and the system config (screen capture from the web server).

I have the event log, but from looking at the web console, I think I have already pushed the actual rebuild event I described above out of the event buffer. Am unsure if there is a convenient way to attach a pdf file or the htm file from the areca web console. (can you recommend a way?) worst case I can capture a bunch of screenshots.

The only activities I have done are:
1) Recreate the RAID6 raidset with different combinations of 64bit LBA/4K blocks, and 64 vs. 128K stripes. The raidset was created without any re-initiatilization to allow for a raidset recovery.
2) Attempted to run the areca check-volume function on the above configs. (in the 3 times I ran that, I got many volume errors after 5-10 mins).
3) Boot up with linux an tried to mount (linux was unable to confirm the type of file system, so mostly left it at that).
4) Run R-studio in demo mode to see what could be viewed. While with the raid setup of 64bit LBA + 128K stripe (what I think was my config), R-studio did detect the ext4 partition. (not sure if that implies that I am on 64bit LBA and 128K stripes? will see about testing with alternate raidset configs and see what a rstudio scan says.).

I haven't run any of the special commands like RESCUE, SIGNAT, LeVeL2ReScUe yet (though was looking curiously at those). Agreed, those low level commands I'm hesitant to do without more guidance from someone more authoritative (like areca, or someone who is familiar with my problem more).

Yes, the drives are still in the same physical locations as before.

Re: the backup.. sadly, not the files themselves but the raw media. I have original media disks to recover with but it would sure be painful to reconvert. Backups of such a large volume (8TB'ish) aren't affordable yet.

BTW, I also did leave a message with areca for help. Given that it's the weekend though, I probably won't be expecting anything until monday/tuesday.

Thanks,
Peter

 

[mwroobel]

Before I make any suggestions, please post some screencaps of your log page-by page so we have a historical record of what you have done. Next, PM Jus and Odditory about this thread, who will also have I am sure excellent suggestions to make once we have more info. Next, download (but do not update until you hear back) the latest firmware for your board. Next, please post a screenshot from the Raid Set Information submenu of the Raid Set Function menu. It may be that a No-Init Rescue or Level2Rescue can fix things, but lets take one thing at a time so as to do no additional harm.

Well, I see Jus and odditory have been to the thread before I got back When it comes to Areca recovery (especially weird failures), Jus is the guy I go to! Follow his lead....

 

[pchiang]

See below for the event log info. Again, the actual data loss event fell out of the buffer (unless there is a way to see beyond what the web console shows).

Thanks for the link to the latest firmware. Have grabbed it, but will hold off on installing.

See below for the raidset details. (am not sure if this is what info/page you are looking for)

So far everything looks normal, but that is probably because I had already recreated it (with no init).

Thanks,
Peter

 

[Jus]

Do you still have that faulty drive in the config?

 

[pchiang]

Hi jus. Thanks for joining.

Yes the flakey drive is in the raidset now. Should I take out or replace w a new one?

 

[Jus]

First of all, YES, you need to get rid of that drive! But not so fast.
Update the F/W and all other filesets along with it.

Since you do not know your RAID set initial parameters, it may take a few tries before you find the right ones.
In order to be able to make another attempt to recover your data, the flakey drive must be replaced with a good one. Having the same number of drives in configuration is essential for successful RAID set signature recovery. So, once a new drive is in place, you will attempt to do another NO INIT RAID set reconstruction. After that the newly added drive must be removed before you try accessing the data.
Removal is necessary for the controller to know it must use parity blocks instead of empty blocks on that new drive.
If you cannot read the data - it means you put in wrong settings.
Once you find right settings, you should be able to access your data.
That "new" drive should not be attached to the controller unless it was formatted elsewhere. If you put it back in the config before formatting, controller is going to see the drive having the same signature as the existing RAID set and attempt to use it. Do not let it happen.
There is a good chance that not the flakey drive is at fault here, but the cable it is hooked up to the controller with. See if you can get it replaced as well.

A word of advice - do yourself a favor - ditch the Samsung drives all together, get Hitachi drives instead. Spinpoints are known for dropping off of RAID sets for no apparent reasons.

 

[odditory]

I haven't run any of the special commands like RESCUE, SIGNAT, LeVeL2ReScUe yet (though was looking curiously at those).

The irony is those commands are less severe than the NO-INIT step you took at the outset, so you leapfrogged the safer options that attempt to recover the raidsig/meta and began with the nuclear option, which overwrote the sector that contains the raidsig/meta. NO-INIT is the last resort AFTER the last resort and requires original driver order + original volume parameters as you've already figured out. And when dealing with a bad drive in the bunch, requires an additional careful sequence of steps as Jus mentions (inserting a blank spare for the creation step but then removing it).

By the way I would not have that raidset online with that faulty drive in there, since you aren't sure if you've matched the original volume properties -- if it drops and rejoins, any further rebuild action with the wrong volume parameters is going to begin overwriting your data with trash much like a zero fill operation would. You are more likely to get your data back if you take this slow and stop trying so many different things in rapid succession. Each one can have consequences that you can't really foresee if you haven't been through it before.

In previous post Jus provided perfect information when it comes to dealing with NO-INIT recreation with a suspect drive. And unfortunately there is no tool (that I'm aware of) that will just auto-detect these volume parameters by looking at the individual disks. When data recovery experts receive a bunch of disks and need to determine things like order and stripe size and block pattern, they're working in an editor like WinHex with one window open per drive, analyzing back and forth among them for many hours. MOST important thing is original drive order. If you have that then you're basically homefree and volume params is just a matter of trying combos until one hits.

 

[pchiang]

hi Jus/Odditory:

Thanks for the advice. Didn't realize that the raidset metadata would still exist (since after the restart of the system/card, the console was reporting no raidset info available. ). Anyways, lessons learned for the next time around.

I have just:
1) Removed the faulty drive from the raidset and plan to swap out the SATA cable.
2) Upgraded the firmware

Working to free up another 2TB drive now to take up the place of the faulty drive, and then will reformat. After that, crossing fingers, will try the no-init raidset recreation again and then see if I can get access to the files.

I did not realize the spinpoints had a bad reputation for raids.

Hopefully will be able to report back later tonight on the status. (If not, I'm out for a week and will tackle this project again).

Thanks again!
Peter

 

[odditory]

As for the "reputation" of Spinpoints in hardware raid I wouldn't worry too much about it, if things have been working okay for you until now aside from the one drive that's consistently dropped. Maybe things are worse with other hardware raid controllers in terms of random drops with Spinpoints but Areca tends to be more tolerant. Actually there was a time when even Hitachi Deskstars - known for the hardware raid friendliness - would occasionally drop from arrays and it had nothing to do with CCTL/TLER but the areca firmware being a little too aggressive and impatient in dealing with an otherwise perfectly healthy drive under some circumstances.

And it was these false-positive timeouts of years past on Areca and other controllers that had people jumping to conclusions about particular drive brands, TLER, etc. But at least in the case of Areca some of the issues that created false positive timeouts/drops have been been rectified in the more recent firmware revs.

So that's why I take reputation with a grain of salt pending my own testing because there are so many variables people dont bother to consider and it keeps the FUD train going. You could very well find that the drive you think is "faulty" passes a full surface scan without errors, and SMART stats show no reallocated sectors. That would point to another variable like cabling as Jus mentioned or even a bad port on the controller (VERY rare but I've had it happen), or the controller board on the drive having a failing or failed component, etc.

 

[Plastikman]

Hoping to put you at ease a little about sammys.

I have 2 RAID arrays on Areca cards with Samsung Drives.
One has 8 Samsung F4's in RAID6
And one has 16 Samsung F4's in RAID6 with a hot spare.
both arrays have been running since 10/21/2010. not a single dropped disk or error.
(both are backed up)

the rest of my arrays are hitachi 5k2000 or 5k3000's also good drives.

(unrelated.. now all the good consumer raid friendly drives are now gone thanks to buyouts.... it makes me sad )

 

[pchiang]

Hi all:

So, made the attempt to recover but thus far no luck.

Here's what I did:
1) Upgraded the firmware, swapped out the cable
2) Grabbed another drive and had it formatted (outside of the raid card). Used that to replace the flakey drive.
3) Recreated the raidset with 8 drives without the init.
4) pulled the sata plug from the new drive, and saw the raidset go into degraded mode. (exactly what was expected/desired)
5) booted up the os (linux) and tried to mount the ext4 volume, but no-go.

I tried this for my most likely combos of:
* 64bit LBA + 128K stripe
* 64bit LBA + 64K stripe
* 4K block + 128K stripe
* 4K block + 64K stripe

As a side note, I ran the r-studio demo, using a scan against the least likely combo (4K block + 64K stripe). it was also able to pick up the ext4 partition, similar to the 64bit LBA/128k stripe situation earlier....

Gotta step away from this for a week (out with the family) but will be back afterwards to continue at this. (also realized we did store the family pics/videos in there , so that certainly increases the importance of trying to recovery at least some data).

I wonder if the fact that the raidset + volume configuration happened to get lost in the middle of a rebuild may have been a factor in preventing a clean recovery? I'm pretty doubtful I would have used a config outside of the combos above, so uncertain on the next steps.

(One thing I have to say - sort of kicking myself for not saving more of the event logs before the got pushed out of the buffer. Any (unlikely) chance there may be more of the event logs available that isn't visible from either the local or the web consoles?

Thanks,
Peter

 

[pchiang]

Hi all:

Just wanted to report a happy ending to this whole ordeal. (whew!).

Found the stripe size, block type (4k vs. 64bit lba), and the correct drive order, and then was finally able to mount and use the drive.

In case it may help others, here's what I did.

The main thing to find was the raidset parameters, since I was relatively confident that the data was still intact. (Even if not fully intact, most was likely still there). The strategy employed was to find a file - in my case it was easiest with a picture file that was 1-2MB in size - and then work out the parameters by changing the raidset config and re-evaluating how that picture looked. Details below:

1) Start out using what I guessed was the most likely. My most likely config was 64bit LBA + 128K stripes.

2) Used rstudio in demo mode to scan the drive (and save the scan too), to find something I could recognize. in this case, I located a jpg that was at least > in size to stripe size*6 (my # of non parity disks in the raidset). I uesd a jpg file that was 2MB in size and previewed the file contents. (you should find a file that is mostly fully rendered).

3) So that I could repeat the operation on a different raidset config, I needed to get the disk offset for that file so I could repeat a much smaller scan near that location. So I cheated a little bit. Was using linux so I ran rstudio under strace (strace -ff -o stracelog -f rstudio &). After that I just tailed the stracelog.*, and clicked on the same jpg file. strace will display the disk offset in the pread system call. You should note that disk offset, as well as the last modified date for that file that rstudio reported (this is so that you can go back to the right file).

4) If the preview is majorly scrambled, then you may need to try a different stripe size. in my case, the stripe size of 128K was in fact correct, so most of the picture rendered properly. I did try 64K and got a lot more scrambling. So that was a clear indicator that 128K was valid.

5) At this point, you use a little bit of deductive reasoning + luck + guesswork. Try different raidset configs, and then use rstudio and start scanning maybe 1GB near that file offset, to find out the impact of your raidset config changes. In my case, the first LBA 64bit + 128K stripe set yielded a mostly clean image, except there was a point where it was obvious a section of the image was swapped out (to me, a clear indicator of an incorrect drive order). Thankfully, the swapped sections were right next to each other, so I ended up swapping 2-3, 3-4, and then 1-2. The last try (swapping 1-2), did the trick, and I ended up with a clean image. Other images in the neighborhood also became clean with that swap.

6) After getting the drive order correct, most/all of the images previewed during the shortened rstudio scan was correct (a great sign). However, the volume still refused to get mounted. The last parameter that I really had not verified - LBA 64bit vs. 4K blocks - was swapped (I changed to 4K blocks). After that, happily the volume mounted properly.

So the final config was 4K blocks, 128K stripes, and the correct driver order (swapping 1-2 from my original setting).

Anyways, just wanted to thank folks for coming to help earlier when the incident first happened. If anybody has similar problems, or has questions about anything I did above, feel free to PM me.

Peter