• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Killed RAID 6 ARRAY - How to recover? Help!

O

okashira

[H]ard|Gawd
Joined
Jul 7, 2005
Messages
1,802
I have 6x WD6400AAKS in RAID6 on a PERC 6/i with alot of data that I need to keep.

I shut down the computer and unplugged one of the sata cables, booted to windows. the perc of course gave me the message of partially degraded.

I shut down again and reconnected and the perc still identified the array as partially degraded. I told it to "import foreign config." and went to windows

Now the drive is GONE.

I went to Disk Management and it tells me I need to "initialize" the disk to use it. I clicked no.

Now the drive is gone from my computer and I cant get to the data.

Idownloaded LSI MSM and it also says degraded array, and when i tried a consistancy check, it told me the array was not initialized....

what can I do?!?
 
Plugged it back in and disk management wants me to initialize it. Should I?

Here what I see in MSM:

13909609.png


I clicked "replace missing drive," and the drive was placed in Slot 6.
I have the option to "rebuild," the drive, i'm guessing I shouldnt do this. I expected a degraded array to be accessible as normal....
Now it started a background initialize automatically. is this bad? :(
 
Last edited:
Your fine, Let your raid array rebuild.

It needs to rebuild because it is degraded.

Your raid array will check the parrity on the drives and rebuild your data like this never happened.


"initialize" is a good thing. When it says that it's asking you if you want to turn the raid array / drive back on.
 
drozenski said:
Your fine, Let your raid array rebuild.

It needs to rebuild because it is degraded.

Your raid array will check the parrity on the drives and rebuild your data like this never happened.


"initialize" is a good thing. When it says that it's asking you if you want to turn the raid array / drive back on.
Click to expand...

Thanks for the post.

Windows Disk Management asked me to and told me that I must initialize the drive. It gives me the option of two partition types "GPT," and one other. It seemed to me that it wants to partition the drive. Is this not bad? Should I let it and select GPT? The drive shows as unpartitioned space.

ill re-add the drive and select rebuild in LSI disk management.
 
Uh, I wouldn't listen to that guy up there.

If windows disk management is asking you to initialize the drive, it means it thinks its unformatted and is attempting to write a new partition table to it. In other words, wipe out your partition tables, which would make data recovery more difficult. Do not try to initialize the uninitialized space.

Now the fact that Windows is even doing this means that your array is already screwed up. When you did "import foreign config" it probably screwed up your raid metadata, which is why windows no longer recognizes your array drive.

I'm not familiar with the Megaraid software or the MegaCLI (commandline interface) so I can't give any specific help there, but it's obvious your RAID metadata configuration is wrong, otherwise Windows would recognize the raid array - I mean that's the whole point of RAID.
 
1. Get this: http://support.dell.com/support/edocs/storage/RAID/PERC6/en/PDF/en_ug.pdf
2. Don't do anything in windows.
3. Reboot and go into the PERC6 bios.
4. Follow the Rebuild instructions in the manual.

FYI if you have a hardware raid controller, you should never let windows try to fix it.. You might be able to save the array if you let it do a consistency check and rebuild itself. Although with Raid 6 you should be able to operate as normal with two drives gone.
 
Last edited:
Yeah, I had a feeling that was the case. I'm thinking i'll

1.) disconnect the drive I originally disconnected, and one other.
2.) power on the system.
3.) PERC should show "degraded"
4.) power off without booting windows
5.) reconnect the drive that wasnt the originally disconnected drive.
6.) power on and import foreign config

the idea here is to get it to set a config that i know should be good / consistant amung 5 drives.

how does this sound?

I will have to flash the old PERC firmware back as when I flashed the newer LSI firmware it broke my ability to get into the PERC bios.


If I can't recover this it will be very dissapointing to know that my simple chain of events was enough to kill a 6 disk RAID 6 array. (not to mention I will be very unhappy with losing about 1.5 TB of data that I cannot recover and wasn't backed up.) this while I was setting up my first WHS.
 
okashira said:
I will have to flash the old PERC firmware back as when I flashed the newer LSI firmware it broke my ability to get into the PERC bios.
Click to expand...

Actually, don't reflash it. You'll have to rebuild it as it is. Use the LSI software, just don't use the windows disk manager.
 
I'm not sure if you will be able to get your data back (I hope you do).

However, this is where you went wrong:

I shut down again and reconnected and the perc still identified the array as partially degraded. I told it to "import foreign config." and went to windows
Click to expand...


The moment you unplug that drive and your controller senses it's gone that drive can never be added back into the array again without rebuilding to it. The reason is that drive now becomes different and is no longer part of the array. Any writes that took place were without that drive, so the data on it is "old" from the array's point of view.

Unplug the drive you messed with and when you boot up one of two things *should* happen.

1) It will just come up and work
2) It will ask you to import the config. Do this. This will import the config that is on the 5 or so drives you have left.

- Then, make sure you have all your data. Back it up to something else if you have the ability.
- Next, shutdown and put the drive back in.
- When the machine boots up do NOT import the foreign config.
- You should still have all your data and MSM should show the single drive as foreign.
- If you have all your data, delete the foreign config from the odd drive and then add it as a hotspare.

The controller should start rebuilding to it automatically.

At least, that's how I do it with my PERC 5/i and 8 x 500GB RE2s.

Riley
 
What do you think about my process? Give it a try or should I rebuild as is?
 
Skud said:
I'm not sure if you will be able to get your data back (I hope you do).

However, this is where you went wrong:




The moment you unplug that drive and your controller senses it's gone that drive can never be added back into the array again without rebuilding to it. The reason is that drive now becomes different and is no longer part of the array. Any writes that took place were without that drive, so the data on it is "old" from the array's point of view.

Unplug the drive you messed with and when you boot up one of two things *should* happen.

1) It will just come up and work
2) It will ask you to import the config. Do this. This will import the config that is on the 5 or so drives you have left.

- Then, make sure you have all your data. Back it up to something else if you have the ability.
- Next, shutdown and put the drive back in.
- When the machine boots up do NOT import the foreign config.
- You should still have all your data and MSM should show the single drive as foreign.
- If you have all your data, delete the foreign config from the odd drive and then add it as a hotspare.

The controller should start rebuilding to it automatically.

At least, that's how I do it with my PERC 5/i and 8 x 500GB RE2s.

Riley
Click to expand...

Posted before I read yours. I think you logic implies I should try to get it to import a config from one of the 5 "hopefully good," drives. I have a bad feeling that when I imported the config from the bad one, it just overwrote the bad config on all the others. lame. :(
 
As a last ditch effort, does the LSI have the ability to create an array without initializing? On most raid cards if the metadata is fubared, what you can do is just delete the bad array data and recreate the raid using the exact same settings (drive order, block size, etc) without initializing, then the volume should just show up in windows. From there you can remove or flag the removed disks as failed and reinsert, rebuild. I know Highpoint, Areca, and Adaptec support this so I kind of assume LSI does as well.
 
Yeah, R-Studio has also saved my butt a few times. Well worth giving a try. I floated the option above as a last ditch effort - you should try data recovery first, which is why you shouldn't initialize the drive like windows is asking you to.
 
Its running R-Studio scan now, all other efforts were fruitless.

If anything the PERC is a performer: on a double degraded RAID 6 array, with a background rebuild going on, R-Studio's still getting a scan rate of close to 100 mb/s.

I took a peek at the r-studio scan results, and it's pretty ugly :( a big mess of files I don't recognize.

letting it scan overnight.
 
Recovering everything is going to take time. Took over a day to scan my entire array and a few more days to actually recover everything.
 
raid-6 on a perc6/i, which is an LSI controller. I can tell you that you should be fine to recover this array without issue. LSI does true RAID-DP (P+Q) parity calcs, which will take more time than say a NetApp RAID-6 (which is much less compute intensive). The LSI implementation is preferred, but rebuilds take longer.

Boy, when I saw this thread title, even though it's not my own array, I gasped and said "oh shit." Glad to hear it's not as bad as I thought.
 
sabregen said:
raid-6 on a perc6/i, which is an LSI controller. I can tell you that you should be fine to recover this array without issue. LSI does true RAID-DP (P+Q) parity calcs, which will take more time than say a NetApp RAID-6 (which is much less compute intensive). The LSI implementation is preferred, but rebuilds take longer.

Boy, when I saw this thread title, even though it's not my own array, I gasped and said "oh shit." Glad to hear it's not as bad as I thought.
Click to expand...

Well, it does't look good. R-studio found somthing like 5 million files on the array and over 14 partitions, both of which are ludacris. I just picked one of the 1.33 TB partitons and started the recovery.

The original array had ~20000 files and one partition.



In other news, the PERC finished the rebuild on one of the drives I disconnected of the data it destroyed. :mad:

Yes, I know it was user error, but, really? :mad:
 
sabregen said:
raid-6 on a perc6/i, which is an LSI controller. I can tell you that you should be fine to recover this array without issue. LSI does true RAID-DP (P+Q) parity calcs, which will take more time than say a NetApp RAID-6 (which is much less compute intensive). The LSI implementation is preferred, but rebuilds take longer.

Boy, when I saw this thread title, even though it's not my own array, I gasped and said "oh shit." Glad to hear it's not as bad as I thought.
Click to expand...

I didn't even know netapps did raid6. I thought they did some sort of weird raid3 type setup (designated parity disks instead of parity spread out on all disks) and had a double parity setup that had two parity disks IE:

Code: 
Volume boot (online, raid_dp) (block checksums)
  Plex /boot/plex0 (online, normal, active)
    RAID group /boot/plex0/rg0 (normal)

      RAID Disk Device  HA    SHELF BAY CHAN  Used (MB/blks)    Phys (MB/blks)
      --------- ------  --------------- ----  --------------    --------------
      dparity   5.64    5     4     0   FC:A  272000/557056000  280104/573653840
      parity    5.48    5     3     0   FC:A  272000/557056000  280104/573653840
      data      5.65    5     4     1   FC:A  272000/557056000  280104/573653840
      data      5.49    5     3     1   FC:A  272000/557056000  280104/573653840
      data      5.66    5     4     2   FC:A  272000/557056000  280104/573653840
      data      5.50    5     3     2   FC:A  272000/557056000  280104/573653840
      data      5.67    5     4     3   FC:A  272000/557056000  280104/573653840
      data      5.51    5     3     3   FC:A  272000/557056000  280104/573653840
      data      5.59    5     3     11  FC:A  272000/557056000  280104/573653840
      data      5.52    5     3     4   FC:A  272000/557056000  280104/573653840
      data      5.69    5     4     5   FC:A  272000/557056000  280104/573653840
      data      5.68    5     4     4   FC:A  272000/557056000  280104/573653840
      data      5.70    5     4     6   FC:A  272000/557056000  280104/573653840
      data      5.44    5     2     12  FC:A  272000/557056000  280104/573653840
      data      5.71    5     4     7   FC:A  272000/557056000  280104/573653840
      data      5.55    5     3     7   FC:A  272000/557056000  280104/573653840

    RAID group /boot/plex0/rg1 (normal)

      RAID Disk Device  HA    SHELF BAY CHAN  Used (MB/blks)    Phys (MB/blks)
      --------- ------  --------------- ----  --------------    --------------
      dparity   5.72    5     4     8   FC:A  272000/557056000  280104/573653840
      parity    5.56    5     3     8   FC:A  272000/557056000  280104/573653840
      data      5.73    5     4     9   FC:A  272000/557056000  280104/573653840
      data      5.57    5     3     9   FC:A  272000/557056000  280104/573653840
      data      5.74    5     4     10  FC:A  272000/557056000  280104/573653840
      data      5.58    5     3     10  FC:A  272000/557056000  280104/573653840
      data      5.75    5     4     11  FC:A  272000/557056000  280104/573653840
      data      5.45    5     2     13  FC:A  272000/557056000  280104/573653840
      data      5.76    5     4     12  FC:A  272000/557056000  280104/573653840
      data      5.60    5     3     12  FC:A  272000/557056000  280104/573653840
      data      5.16    5     1     0   FC:A  272000/557056000  280104/573653840
      data      5.32    5     2     0   FC:A  272000/557056000  280104/573653840
      data      5.61    5     3     13  FC:A  272000/557056000  280104/573653840
      data      5.77    5     4     13  FC:A  272000/557056000  280104/573653840
      data      5.17    5     1     1   FC:A  272000/557056000  280104/573653840
      data      5.33    5     2     1   FC:A  272000/557056000  280104/573653840

    RAID group /boot/plex0/rg2 (normal)

      RAID Disk Device  HA    SHELF BAY CHAN  Used (MB/blks)    Phys (MB/blks)
      --------- ------  --------------- ----  --------------    --------------
      dparity   5.18    5     1     2   FC:A  272000/557056000  280104/573653840
      parity    5.34    5     2     2   FC:A  272000/557056000  280104/573653840
      data      5.36    5     2     4   FC:A  272000/557056000  280104/573653840
      data      5.21    5     1     5   FC:A  272000/557056000  280104/573653840
      data      5.37    5     2     5   FC:A  272000/557056000  280104/573653840
      data      5.22    5     1     6   FC:A  272000/557056000  280104/573653840
      data      5.25    5     1     9   FC:A  272000/557056000  280104/573653840
      data      5.23    5     1     7   FC:A  272000/557056000  280104/573653840
      data      5.39    5     2     7   FC:A  272000/557056000  280104/573653840
      data      5.24    5     1     8   FC:A  272000/557056000  280104/573653840
      data      5.40    5     2     8   FC:A  272000/557056000  280104/573653840
      data      5.29    5     1     13  FC:A  272000/557056000  280104/573653840
      data      5.41    5     2     9   FC:A  272000/557056000  280104/573653840
      data      5.19    5     1     3   FC:A  272000/557056000  280104/573653840
      data      5.35    5     2     3   FC:A  272000/557056000  280104/573653840
      data      5.20    5     1     4   FC:A  272000/557056000  280104/573653840

    RAID group /boot/plex0/rg3 (normal)

      RAID Disk Device  HA    SHELF BAY CHAN  Used (MB/blks)    Phys (MB/blks)
      --------- ------  --------------- ----  --------------    --------------
      dparity   5.26    5     1     10  FC:A  272000/557056000  280104/573653840
      parity    5.42    5     2     10  FC:A  272000/557056000  280104/573653840
      data      5.27    5     1     11  FC:A  272000/557056000  280104/573653840
      data      5.43    5     2     11  FC:A  272000/557056000  280104/573653840
      data      5.28    5     1     12  FC:A  272000/557056000  280104/573653840

I am far from a netapp expert though
 
I think everything was destroyed when the controller initialized the array after re-connecting the drive.

Generally, "initializing" when dealing with RAID is a destructive operation and means "I'm formatting your array and making it ready to use".

"Rebuilding", "re-sync", "mirroring" is good and means it's dealing with data you already have.

Riley
 
Well surely if you spent money on a raid setup for about 2Tb worth of data, you implemented a backup. Why not just build the array fresh and then copy all the data over?
 
NetApp does RAID-4 or RAID-DP, period. Default is RAID-DP, which they refer to as RAID-6 (even though it's not) in a lot of documentation (technical and marketing, both). You're right that they designate disks in the aggregate to be parity and double parity disks. When a parity recalculation occurs, unlike true RAID-6 (where the DP stands for Diagonal Parity), on NetApp, only the non=parity disks participate in the rebuild operation. In (for instance) LSI's implementation, there are actually two parity calculations done on every write, and the data and the parity sets are striped across all disks in the volume. This offers much better rebuild times, but takes significantly more resources to accomplish.

NetApp does a lot of things, however, it's hard to say that it does any one particular thing well. It's well suited for organizations that need a lot of functionality, and they want it in one box... and they don't mind getting raped on the support contract (much like EMC... Evil Machine Corporation). :)
 
Thanks to

alamone
Keiichi
Skud
Blue Fox
sabregen

and (even if the advice wasnt 100% accurate! :))

drozenski

I Was able to get about 99.9% recovery.

This all happened as I was seting up my first backup solution for the first time in my 14+ years of computer geekdom. My WHS

R-Studio found it all, and it took about 70 hours to transfer it to my WHS over 100mbit. Alls been compuied back now.


To celebrate, I converted my array to RAID 5. :D
85817156.png
 
glad to hear you got it back. data loss is never cool. i hope you assigned a hot spare!
 
Last edited:
You must log in or register to reply here.
Back
Top