• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

RAID 5 Array Down

F

fibroptikl

Supreme [H]ardness
Joined
Mar 9, 2000
Messages
7,339
I have a HighPoint 464 RAID Card that runs in my old computer, a P3 650 w/ 512MB RAM, 80GB boot drive.

On the RAID card are 8 Western Digital 200GB ATA100 8MB drives. I was playing some music (from the RAID array) earlier today when I was burning a DVD and it stuttered - considering that happens I didn't think much of it. I was playing a song that was with in a large file, and it was slow. But that happens, and given the fact I was burning a DVD - my songs eventually played, and I didn't listen to any music after 2 or 3PM at the latest.

Until tonight - I went to play a file (ironically the same file) and it didn't come back up.

I went upstairs, and came down and it had a Win32 error (in foobar2000) and I I RDP-ed into the server, and found there was a Delayed Write Failure. Opened Explorer; no M: drive (array). Opened Disk Management - no M:\. (Or unknown devices that could have been the RAID array).

Went to the back storage closet where I keep it and I start to hear it: BEEP BEEP BEEP BEEP BEEP. Loud, but not that loud to hear it where I am located. Shut down, make sure everything is connected. I boot into the RAID Card's BIOS and look around. I see a list of drives, and I hit "Continue" (to just ignore) and it gives me another screen. I hit Power off. I do it again, and I realize this: One array has 6 drives, the second has 2.

So 2 drives have failed on a RAID 5. It's now shown as two seperate array's.

So I have questions on how this happened: A) Why did my hot spare not take effect? It's possible it did (in the last day or two) and I didn't notice it. and B) Why wasn't I notified? I set the software to notify me, and I test it quite frequently.

I'll say this I have a backup. However, I'd like to try and restore the data (worst case, it doesn't work and I continue on). I am thinking of trying a new power supply, cause the one I had while enough power (only powered the drives) - might have gone bad. However, the array is already in two seperate arrays, and they can be seen so the drives have to be getting power.

I'm thinking of trying the GetDataBack RAID Reconstructor - anyone ever use this? From there, I'll have to go about creating an extra 1TB of space (to put the image/data to).
 
The hotspare may have taken effect, however, it's not an instantaneous "poof the array is up" when the hotspare is added the array still needs to rebuild just like if you powered down and replaced the drive. So if another drive dies during the swap, your array is gone.
 
fibroptikl said:
Western Digital 200GB ATA100 8MB drives
Click to expand...
There's the problem. Those are SE drives, not RE. TLER problems, etc, are bound to happen.
fibroptikl said:
So I have questions on how this happened: A) Why did my hot spare not take effect? It's possible it did (in the last day or two) and I didn't notice it. and B) Why wasn't I notified? I set the software to notify me, and I test it quite frequently.
Click to expand...
Do you have a log of when disks drop and are re-added? I would guess you've been dropping and rebuilding fairly often (once a week, or thereabouts). This time you just happened to drop two disks at the same time.
 
The 2 array thing is pretty interesting.. It looks like the hot spare became part of an array. Looks like the controller fouled up. Not sure you could do anything to recover without a restore. You might try disconnecting the drives then boot into the array config. and clear the controller. Then reconnect all but the hot spare and see what config. is written to the drives. I have a feeling that since it is set as 2 arrays, that is what will be on the drives as well. I've never seen an array split into 2 like that on its own. Is it reporting any of the connected drives as bad, or just the arrays are missing disks?
 
Aronj66 said:
The 2 array thing is pretty interesting.. It looks like the hot spare became part of an array. Looks like the controller fouled up. Not sure you could do anything to recover without a restore. You might try disconnecting the drives then boot into the array config. and clear the controller. Then reconnect all but the hot spare and see what config. is written to the drives. I have a feeling that since it is set as 2 arrays, that is what will be on the drives as well. I've never seen an array split into 2 like that on its own. Is it reporting any of the connected drives as bad, or just the arrays are missing disks?
Click to expand...
Nope, reports all drives as available, and it doesn't say anything about an array missing a disk.

It does however see them as two seperate, yet broken arrays.
 
eventsel3.jpg


I'm not sure if the 2nd channel is the 2nd or 3rd (depends if they start with 0 or 1), but two drives, on the same channel failed. I'm thinking it was caused by the power supply. I'm tempted to see if I give the drives power by a different power supply, it might come up?
 
You are absolutely right that it is probably the power supply. I have seen this happen twice with Adaptec cards. Disconnect everything that is not absolutely necessary to use and reboot into the bios. If that solves the problem then get a new power supply and don’t use the computer until you do replace it. You may want to replace the power supply anyway.
 
The box uses dual power supplies - I'm thinking of replacing two of them with one now.

Problem is finding an older ATX power supply that has a nice load that can power 9 drives.
 
Rembrandt said:
You are absolutely right that it is probably the power supply. I have seen this happen twice with Adaptec cards. Disconnect everything that is not absolutely necessary to use and reboot into the bios. If that solves the problem then get a new power supply and don’t use the computer until you do replace it. You may want to replace the power supply anyway.
Click to expand...
I went ahead and moved the system over to my side of the basement (where I can work on it better) - I put all new cables on it and tried it - no luck. I grabbed two molex connectors from the other power supply and the drives still stayed offline.

So I'm still not too happy. The backup drives are missing a few unimportant music files - but I'd still love to get them.

I'd hook up those two drives to see if my computer see's them - but I don't want them being modified (drive signatures, misc data, etc)
 
If I understand correctly, there was only one array. If so, the problem is the disk signatures are messed up on the 2 drives. If one drive was a hot spare, then the original array should only be missing 1 drive? I would remove the 2 drive array in the controller, wipe the drives and try to reintroduce them into the large array and see if it can rebuild. Or there have been posts about software that can recover data from broken arrays, you could do a search and try that. Power issues may have caused the problem, but now the disk signatures are fouled up and just fixing the power will not change that.
 
Whatever you do, if you start to play around with the drives (disconnecting them from the controller and testing them) Make sure you know exactly which port they were plugged into.

You could always try this...it has worked for me on a failed raid 5 array that had a bad drive and then proceeded to take a dump.

Put the drives back in as they were, with the questionable drives included as well. Then go into the controller setup and delete the array (scary I know). Then, recreate the array, including all of the drives that were originally included, same stripe size, keep everything the same as the original array. Then when it comes time to finalize, do a quick init (your controller may call this option something different). Basically, its a non destructive rewriting of the arrays metadata. So it leaves the drive contents as is, and just rewrites everything pertaining to "you are drive 1, you are drive 2, etc..you are a member or Array X...yada yada"

Then save the config and reboot.

Like I said, I have done this and it has worked, in fact I have done this three times and each time it worked. Of course it was with either a known bad drive, or a quesionable drive that kept disconnecting from the array. If it does work, then get the stuff off you want and then try to figure out what the cause was behind what happened.
 
I would do that, but I'm not sure if my RAID controller supports it - and there is only one way to find out. (Well I could look in the manual or e-mail tech suport)

However, I'd like to try some simpler stuff first, like a new power supply, or freezing the hard drives.
 
It seems that a drive was dropped a bit too prematurely.

After working with HighPoint for a week or so (mainly cause I get one response a day at 12PM or so) I was able to bring back the array. Data was still intact. There were a few write delay failures on two folders, and a chkdsk /v /f took care of that. I'm going to be running a Verify once I hear back from them (and I wanted to shut down the PC cause it's loud and I wanted to attach the fans).

But it's up.

Lesson learned: Use quality hardware.
 
You must log in or register to reply here.
Back
Top