Testing failure & rebuild

[tarnar]

Hey all,

I've been working on my storage server and am getting very close to actually considering it 'ready for use'. The last thing I want to test is that resilience actually works and see how long a rebuild actually takes.

A quick overview of the setup.

ESXI 5.1 server
- Xeon
- 16GB ECC RAM
- boots from SSD attached to the motherboard

FreeBSD guest
- passed-through Dell Perc H310, cross-flashed to LSI IT mode
- 2x four port SATA breakout cables w/ sideband to enclosure
- 8x 3TB Seagate SATA drives
- 12 GB RAM

zpool created as the equivalent of a RAID60
- 2x four disk raidz2, striped together
- designed so that a single port failure on the HBA is survivable

Now, I'm not going to test the port failure, but I'm definitely keen on testing a single disk failure and the rebuild time.

The eight disks are in hot-swappable trays.

The sensible bits of my brain tell me that just yanking a drive to test the resiliency is a bit of a bad idea. If for no other reason than I'd like the disk to still work afterwards, and unplugging a hot disk without any prep (the least of which is to spin it down) seems like a bad idea.

I figure I need to do this in steps.

Step 1, 'fail' the disk in the zpool
Step 2, tell FreeBSD to stop talking to the disk
Step 3, actually unplug the disk!
Step 4, zero out said disk on a different machine
Step 5, plug the disk back in
Step 6, re-add it to the zpool
Step 7, monitor the rebuild

Are these steps valid? Am I missing obvious ones? Am I making steps up where none are needed?

What specific commands do steps 1, 2, 5, and 6?

Another big question - does used space impact rebuild times? Do I need to fill the zpool with data to get a realistic rebuild time?

I do know which disk is which in the machine, in terms of da# and which physical drive that da# corresponds to. I have used the LSI provided SAS2IRCU program to examine the disks in the enclosure.

Thank you in advance for any assistance or advice offered.

 

[madrebel]

zpool created as the equivalent of a RAID60
- designed so that a single port failure on the HBA is survivable

your disks are directly connected to the HBA then?

The sensible bits of my brain tell me that just yanking a drive to test the resiliency is a bit of a bad idea. If for no other reason than I'd like the disk to still work afterwards, and unplugging a hot disk without any prep (the least of which is to spin it down) seems like a bad idea.

in pre prod, don't be squeamish. This is no time to be sensible lol.

Step 1, 'fail' the disk in the zpool

failing a disk in ZFS will require you to wait for 127 transactions groups to pass before you get any reasonable rebuild. withing that 127 TX group window you'll do a quick rebuild using the data from the previous transaction groups which are still on disk ... i may have this wording incorrect but there is a 'roll up' of sorts that can occur if the 'failed disk' returns to the system quickly.

easiest way to do this is just pull the disk or zpool 'pool' offline cXtXXXXX.

the disk will still be spinning when you pull it btw, even if you offline it first. you would have to have other tools that I'm not aware of to cut the power from the disk before pulling it.

Step 2, tell FreeBSD to stop talking to the disk

zpool 'pool' offline cXtXXXXX

Step 3, actually unplug the disk!

again, the disk platters will still be spinning when you yank it. don't be alarmed. when i pull a disk i run devfsadm -Cv to clean up the old link.

Step 4, zero out said disk on a different machine

unnecessary. just zero out the MBR to destroy the ZFS config data.

Step 5, plug the disk back in

spit on it and push ... after you do this you want to to another devfsadm -Cv not 100% if freeBSD has the devfsadm tool. omnios is either devfsadm -Cv or nexenta from nmc lunsync -r (can do devfs from root/bash too)

Step 6, re-add it to the zpool

so you can't directly do this. your zpool will show that a drive is missing so to fix that you need to know the devid from your new drive and do a

zpool 'pool' replace -o cXtXoldlunid -n cXtXnewlunid (-f to force if need be)

to get a list of stuff your system sees either use iostat -en or sasinfo logical-unit .. again those are solarisy commands, not 100% on freeBSd. once you have the current list of what your system sees compare it to zpool status 'pool'. once you find the new disk ID you can quickly verify with a zpool status 'pool' |grep cXtXnewid.

Step 7, monitor the rebuild

zpool status 'pool' |head

you'll see the times for the resilver. the first couple minutes this time will radically fluctuate if you have a LOT of data and a busy system. give it a few to stabilize.

Another big question - does used space impact rebuild times? Do I need to fill the zpool with data to get a realistic rebuild time?

yes and yes. dd if=/dev/zero of=/path/tp/volume/bigassfile.out bs=1024 count=alot .... use idk, whatever gets you to say 60% full as the count and let it run.

I do know which disk is which in the machine, in terms of da# and which physical drive that da# corresponds to. I have used the LSI provided SAS2IRCU program to examine the disks in the enclosure.

there has to be better tools than that.

see if devfsadm, iostat, sesctl, and sasinfo exist. i need to freshen up my bsd, been awhile.

 

[tarnar]

your disks are directly connected to the HBA then?

Yes.

there has to be better tools than that.

see if devfsadm, iostat, sesctl, and sasinfo exist. i need to freshen up my bsd, been awhile.

camcontrol(8) seems to be what I'm after for the spin-down/disconnect.

First, to fill the disk with junk.

Instead of one really really big file I'm rolling out lots of 4 gig files. Some from /dev/urandom, some from /dev/zero. Some copies, some cat'd together.

The rest makes sense. Once the pool is relatively full I'll get dangerous.

This might take a while. Write speeds, multi-threaded, are roughly 150 megs/sec.

 

[TeeJayHoward]

SATA/SAS disks are designed to be pulled while still running. The physical connector cuts the power in the "right way". If I recall correctly, the heads even get parked. Don't worry about it, just pull the disk.