Raid 5 - What to do when 1 drive fails

[defenseman]

Basically, my question is: When one drives fails in a Raid 5 configuration, what should happen?

Now, yes I know I should replace the failed drive. But what happens after that? Should I check for consistency? Rebuild the array? How much time do I have until another drive can fail?

What I'm trying to do, is do a hasty backup of my system to another system. I have multiple systems that are exactly the same including a 3 disk Raid 5 system on each. I'm hoping I can somehow copy my system by replacing 1 drive at time with my good system. Can this work?

[axan]

Basically, my question is: When one drives fails in a Raid 5 configuration, what should happen?

Now, yes I know I should replace the failed drive. But what happens after that? Should I check for consistency? Rebuild the array?

when a drive fails in raid 5 and is replaced with new one, the array will automatically rebuild itself (well 99.9% of the time, in case it doesn't just run rebuild process) depending on size of array this can take from hours to days.


How much time do I have until another drive can fail?

There is no fixed amount of time, every minute you keep your damaged array online you risk another drive failure and data loss


What I'm trying to do, is do a hasty backup of my system to another system. I have multiple systems that are exactly the same including a 3 disk Raid 5 system on each. I'm hoping I can somehow copy my system by replacing 1 drive at time with my good system. Can this work?

Yes this will work, replace 1 drive at a time and wait for the array to fully rebuild itself before swaping another drive

[defenseman]

Ok then. In my raid configuration utility, I can force a rebuild. Should I do that? Right now I'm checking for consistency and seeing if that will "rebuild" the data.

[nitrobass24]

What I'm trying to do, is do a hasty backup of my system to another system. I have multiple systems that are exactly the same including a 3 disk Raid 5 system on each. I'm hoping I can somehow copy my system by replacing 1 drive at time with my good system. Can this work?

Yes it will work, but would it not be easier/faster to do do a backup image to another drive and then just restore the image on the new system? This way you dont have to wait around for the array to rebuild itself everytime you pull a drive?

[defenseman]

Yes it will work, but would it not be easier/faster to do do a backup image to another drive and then just restore the image on the new system? This way you dont have to wait around for the array to rebuild itself everytime you pull a drive?

I know I wanted to do that at first, but I'm not 100% linux savvy. I researched using a "dd" but I'm not positive on how to get all the paritions over. I'm not even sure what /dev/sdX to use. Browing my /dev/ folder, I see: sda, sda1, sda2, sda3, and sda4.

[Stoly]

Now, yes I know I should replace the failed drive. But what happens after that? Should I check for consistency? Rebuild the array? How much time do I have until another drive can fail?

The array is re-built automatically, (or should be, if it doesn't there could be a problem with the raid controller)

Are you willing to risk loosing your data? probably no, then you should replace the drive ASAP. If time is a consideration, backup your data.

There's no way to know for sure when a drive is going to fail, keep in mind that all the drives in the raid have the same stress so when one fails the probability that another one also fails increases.

[defenseman]

Update:

After my raid utility checked for consistency, it said "FIXED". I rebooted to check if it would load the OS, and now, right after the raid volume is checked during bootup, it just says "Grub". No input responds.

[nitrobass24]

seems that it did not rebuild the boot sector correctly, i would search for how to fix linux boot fixes or how to reinstall grub and add your os to grub. Sorry i cant help more im not a linux guru myself.

[RavenD]

What distro are you running? I might be able to help walk you through it.
I this is why people tend to recommend running a dedicated boot drive...

http://www.hardfolding.com/ftag1.php/mem/8591/1/0.png (http://www.hardfolding.com?go=38&tnum=33&id=8591)

[axan]

you should run a small hdd as a boot drive, booting from raid is asking for trouble, specially if you're linux n00b

[defenseman]

I'm using Fedora 9.

I'm not too concerned because I actually didn't lose any data (see org post). I'm testing out a method of creating an image of current system.

If anyone recommends a way to image a linux system, please, please post it. I've spent all day reading about "dd" but it goes for every byte...which sucks.
Note:
- I don't have a single drive bigger than 36Gb...not even an external drive.
- My system hard drive (sda) is about 69Gb in size, but definitely using less than 10Gb.
- I have two other systems, each with their own Raid 5 system that both eventually need the image of system 1.

[nitrobass24]

you should run a small hdd as a boot drive, booting from raid is asking for trouble, specially if you're linux n00b

Not if your running a good backup

[bexamous]

What? What OS are you running and how did you do the install? Since when can Linux have /boot on a software RAID5? Unless something changed recently that's never worked.

I don't even know any installers that don't screw up putting /boot on a RAID1... usually they only install grub on a single drive rather than on each drive.

If you want to reinstall grub boot off a LiveCD... run grub and do 'root (hd0,1)' and then 'setup (hd0)' for example. This would be if /boot was /dev/sda1. If you have /boot on a RAID1 over two drives you would acutally want grub on both drives.... if /dev/sda1 and /dev/sdb1 were in RAID1 you would run grub and do:
root (hd0,1)
setup (hd0)
root (hd1,1)
setup (hd1)

I think this is right? Or maybe its
root (hd0,1)
setup (hd0)
setup (hd1)
because if either drive fails the single remaining /boot would no matter what be (hd0,1)

Anyways I dunno.

[defenseman]

I never said it was software raid! Its on a hardware raid controller.

[bexamous]

Oh sorry in that case it doesn't really matter what you're doing.

If you have two identical systems, boot off a LiveCD on each system (Ubuntu Owns!), make sure they both get an IP and all and network is working...

On the system you want to write data to, the 2nd system / backup system whatever you want to call it run:

nc -l -p 50005 | dd of=/dev/sda

This is assuming you have only you single array, otherwise you might have to change /dev/sda to whatever the array is. You can use 'fdisk /dev/sda' and hit 'p' and it'll print out some crap including size of the drive... then 'q' to quit. Easy way to identify what sda is for example.

Anyways....

On your main system....

dd if=/dev/sda | nc 192.168.0.2 50005

And obviously change that IP address to whatever the first system has.

This will byte-by-byte copy the data on the array from the first system to the second system.

After this you might want to restart the 2nd system and try to let it boot to the OS... just incase it works. If it fails for any reason boot back to the Ubuntu CD.

There is two files you might have to edit:

/boot/grub/grub.conf and /etc/fstab

The reason is because if the OS uses disk-ids or UUIDs or anything unique to identify a harddrive it will not be correct. You'll need to fix that or just change them to /dev/sda1 or whatever.

You should not need to reinstall grub or anything like that.

Also just to note this will not work very well if the drives are not the same size. Eg one system has a 500GB array and the other system is only 300GB... not gonna work.

In that case you would likely want to boot to the 2nd system, partition out the array however you like... format the partitions in a similar manner... and then mount them in the right way... eg if you have:
/dev/sda1 = /boot
/dev/sda2 = swap
/dev/sda3 = /

You would then do:
mkdir /mnt/system2
mount /dev/sda3 /mnt/system2
mkdir /mnt/system2/boot
mount /dev/sda1 /mnt/system2/boot

Then to copy over your old system you would use rsync... I'd boot system1 off a LiveCD to make things easy so you don't have to skip /dev/ and such... anyways on system1 mount the OS similar to above and then to run rsync:

(if run on system1)
sync -av /mnt/system1/ 192.168.0.2:/mnt/system2/

That'll copy everything over. You'll have to however install grub since....

grub
> root (hd0,1)
>setup (hd0)
> quit

Thats the jist of it.

[Lazn_Work]

actually pulling one drive at a time isn't a good way to clone the system.. Because data on the drives change between replacing the first and pulling the 2nd, then again for the 3rd.. so by the time you are done, the 3 drives that are out of the system are out of synch.