ZFS lockup with hard drive failure

P

planum

n00b
Joined
Mar 10, 2011
Messages
17
I searched for this on the web and the closest thing I came up with was a 2009 discussion about a bug in opensolaris being the culprit but I am running openindiana 148 and I expect it has been addressed. I have a 16 bay sas enclosure populated with 2tb SATA drives (hitachi x7) connected to server via 8088 cable to lsi 9200 HBA card. I bought a new hard drive to add to the system rebooted and created the raidz from scratch with the new drive added. Initally no problems but ran bonnie++ and new drive light stayed on continuously zpool stopped responding even though there drive passes quick test and no smart errors. When i created a zpool raidz with the 7 drives at first it was fine then after running bonnie the single drive light went on and stayed on.
I did a zpool status -x and the command hangs up never displaying the drive list so that one can see which drive is corrupt.
At any rate, my question is this: I had a tip off from the first few minutes that I created the new zpool raidz with the new drive that it was defective, but what if it failed 30 days from now and I didn't know which drive failed. When I run zpool status and it hangs up and doesn't tell which drive is degraded/unavailable how would I be able to query solaris for information on which drive in the raidz failed so the drive could be changed?
Thanks for any advice.
 
Did you check the system logs? /var/adm is the location in solaris, have not run OI yet but assuming they are in the same place. If it is a driver issue I would except a couple of lines to show up in those logs.

I have thrown a couple of drives into a few mirrors I knew were bad just check the operation of controllers - M1015 used in JBOD mode with IR firmware. Never had a lockup like that before.

Have you stress tested the enclosure before with the current 8088 cable you are using?
 
iostat -eEi will yield the drive error statistics and the -i option generates the device id so that you can correlate this to the zpool status output if you can get that. However, if the zpool status -v command hangs after a few lines of output and you cannot see the individual drive states/status of the zpool will you risk accidentally destroying the raid if you just remove the drive iostat is saying has errors without using the zpool commands to do so.
 
No, it should not destroy the pool by just removing the disk. The pool should go to degraded if you pulled the failing drive. It should switch to offline if you pulled the wrong drive and are running raidz1.
 
validation of the hardware is the plan as this is a new enclosure with new drives.
btw what do you use for stress test? have been investigating the sunvts as an option.
 
yes that is partly my question, let me show you the /var/adm/message part:


Jul 29 16:50:36 lyra fmd: [ID 377184 daemon.error] SUNW-MSG-ID: ZFS-8000-GH, TYPE: Fault, VER: 1, SEVERITY: Major
Jul 29 16:50:36 lyra EVENT-TIME: Fri Jul 29 16:50:36 EDT 2011
Jul 29 16:50:36 lyra PLATFORM: ProLiant-DL380-G5, CSN: USE750N0VJ, HOSTNAME: lyra
Jul 29 16:50:36 lyra SOURCE: zfs-diagnosis, REV: 1.0
Jul 29 16:50:36 lyra EVENT-ID: f3b7c640-2bb9-47f6-f599-ea277dbb3e3b
Jul 29 16:50:36 lyra DESC: The number of checksum errors associated with a ZFS device
Jul 29 16:50:36 lyra exceeded acceptable levels. Refer to http://sun.com/msg/ZFS-8000-GH for more information.
Jul 29 16:50:36 lyra AUTO-RESPONSE: The device has been marked as degraded. An attempt
Jul 29 16:50:36 lyra will be made to activate a hot spare if available.
Jul 29 16:50:36 lyra IMPACT: Fault tolerance of the pool may be compromised.
Jul 29 16:50:36 lyra REC-ACTION: Run 'zpool status -x' and replace the bad device.
Jul 29 16:50:48 lyra syslog[903]: [ID 702911 daemon.warning] WARNING: Failed to send buffer
Jul

So if the zpool status -x command fails, what is plan B?
 
Have you rebooted the box? Is there any chance something is writing to the faulted pool? Is it the root pool?
 
how hard were you running the stress tests ? Nexenta reported (and we have had it happen twice), when you pound on a set of sata drives connected to SAS chain, not sure if its the sas expander or something to do with the sata signalling, but ALL the drives go offline (or in error). Luckily we recovered ok, but it has happened when we were resilvering a raid set. ..

Nexenta has something on it but I don't know if it carries over to OI or SE11. Not sure your would be related to it, but if you're stress testing, could be if you have sata drives on a sas expander chain.
 
was not to the point of stress testing yet. This was a bad drive right out of the box. Zpool was in degraded state shortly after it was created and it flagged this drive. I removed it to test it on another computer (hitachi drive diagnostic)-- which it passed. So I thought I would give it another try. I destroyed the pool and replaced the drive and rebooted, etc. Then next time I tried to create a pool with that drive in it, the light stayed on forever and the zpool status command hangsup after printing out the first couple of lines (but never gets to point of printing out the drive list/status). OI wouldn't even let me kill -9 the zpool process. So I removed the drive put it back in other PC and ran the advanced Hitachi diagnostic Test and it failed (oddly it said it was a non Hitachi drive and no explanation was given for the failure--even tho it is a deskstar drive). This was being use on data pool not rpool.
 
Spent a good deal of time today with this glich and so I want to summarize a few points that may save others time.
If a drive is bad and the zpool status command hangs, it seemed to resolve when you physically remove drive and try again
Using cfgadm -al to get the id of the bad drive; then cfgadm -c unconfigure id gives you message about not being a valid attachment point-->you can try devfsadm -C but that didn't work for me and fmadm fault -a told me there was a faulted device but never identified it and wanted me to run zpool status -x which would hangup. After alot of trial and error and googling I could never resolve why cfgadm's list of ID's wouldn't pass through the cfgadm -c configure without saying invalid ID ineffect. Ended up rebooting all the links to the bad drive were still there and only way I finally removed them was that AFTER the reboot the devfsadm -Cv got rid of all the traces of the failed drive. I did even try zpool remove command and zpool destroy. I hope I'm not making this more confusing to anyone reading and I hope this is a rare problem that may have only happened because I did something out of sequence, etc. If anyone has dealt with this and could summarize the steps for taking a failed harddrive out of the array/pool that would be greatly appreciated--especially if I have screwed up the explanation/proper sequence and confused other readers.
 
You must log in or register to reply here.
Back
Top