![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
So i've got an annoying problem with one of my OI+Napp-IT boxes.
The box is a very large pool used for storage of backup archives. We're using rather low quality drives pretty hard, so failures are relatively common, but this is the first issue i've had replacing a disk. For reference, this is a 72-drive beast - 70x 500gb in 7x10disk RAIDZ2's in the pool in question.
First, the disk was throwing a lot of Hard and Transfer errors, slowing down the server. It never failed in ZFS' eyes (this is very common on this box, I lose 1-2 disks a month in this fashion). I popped the disk out, as I have in the past, causing the RAIDZ2 to degrade. I replaced the disk in the bay and went to do a replace with Napp-IT.
Napp-IT found the new drive, but did not have the old/failed/removed drive in the list of drives you could choose to replace (this hasn't happened before). I tried adding the new drive to the pool as a spare (with Napp-IT). The pool did not use it - I tried a few things like telling the pool it was fine, hoping to kickstart a rebuild with the spare - no luck. I removed the spare with Napp-IT and went to the CLI. I did a "zpool replace backups olddisk newdisk" -- This kicked off a resilver as expected - however, when the resilver finished I ended up with this:
pool: backups
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: resilvered 323G in 30h32m with 0 errors on Thu Sep 6 19:41:19 2012
config:
...
raidz2-4 DEGRADED 0 0 0
c2t5000CCA597D1794Ed0 ONLINE 0 0 0 500.11 GB Hitachi HTS54505
c2t5000CCA597D17CF1d0 ONLINE 0 0 0 500.11 GB Hitachi HTS54505
c2t5000CCA597D17DDCd0 ONLINE 0 0 0 500.11 GB Hitachi HTS54505
c2t5000CCA597D186A7d0 ONLINE 0 0 0 500.11 GB Hitachi HTS54505
c2t5000CCA597D18C3Bd0 ONLINE 0 0 0 500.11 GB Hitachi HTS54505
c2t5000CCA597D18EEDd0 ONLINE 0 0 0 500.11 GB Hitachi HTS54505
...spare-6 DEGRADED 0 0 0
.... c2t5000CCA597D192B5d0 FAULTED 0 0 0 too many errors 500.11 GB Hitachi HTS54505
.... c2t5000CCA597D1BC76d0 ONLINE 0 0 0 500.11 GB Hitachi HTS54505
c2t5000CCA597D036A8d0 ONLINE 0 0 0 500.11 GB Hitachi HTS54505
c2t5000CCA597D19759d0 ONLINE 0 0 0 500.11 GB Hitachi HTS54505
c2t5000CCA597D19C32d0 ONLINE 0 0 0 500.11 GB Hitachi HTS54505
...
spares
c2t5000CCA597D1BC76d0 INUSE currently in use 500.11 GB Hitachi HTS54505
Sooo, it appears that Napp-IT didn't remove the spare (I can't say I verified it actually removed it when i asked it to). The spare DID resilver when I did it manually, however it's just stuck there, showing degraded with the removed drive still FAULTED.
I'm open to suggestions at this point. I've tried everything I can think of (while off-site) to remove the issue short of reboot. I can't remove the spare now because it's INUSE. Clearing the status does nothing.
My next thought is going onsite, pulling the new drive, trying to remove the spare again, and putting it back in... if it doesn't change much I'll try to use the CLI to do another resilver. Failing that the only option I see is reboot and cross fingers.
Anyone else experienced this with an amicable outcome?
The box is a very large pool used for storage of backup archives. We're using rather low quality drives pretty hard, so failures are relatively common, but this is the first issue i've had replacing a disk. For reference, this is a 72-drive beast - 70x 500gb in 7x10disk RAIDZ2's in the pool in question.
First, the disk was throwing a lot of Hard and Transfer errors, slowing down the server. It never failed in ZFS' eyes (this is very common on this box, I lose 1-2 disks a month in this fashion). I popped the disk out, as I have in the past, causing the RAIDZ2 to degrade. I replaced the disk in the bay and went to do a replace with Napp-IT.
Napp-IT found the new drive, but did not have the old/failed/removed drive in the list of drives you could choose to replace (this hasn't happened before). I tried adding the new drive to the pool as a spare (with Napp-IT). The pool did not use it - I tried a few things like telling the pool it was fine, hoping to kickstart a rebuild with the spare - no luck. I removed the spare with Napp-IT and went to the CLI. I did a "zpool replace backups olddisk newdisk" -- This kicked off a resilver as expected - however, when the resilver finished I ended up with this:
pool: backups
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: resilvered 323G in 30h32m with 0 errors on Thu Sep 6 19:41:19 2012
config:
...
raidz2-4 DEGRADED 0 0 0
c2t5000CCA597D1794Ed0 ONLINE 0 0 0 500.11 GB Hitachi HTS54505
c2t5000CCA597D17CF1d0 ONLINE 0 0 0 500.11 GB Hitachi HTS54505
c2t5000CCA597D17DDCd0 ONLINE 0 0 0 500.11 GB Hitachi HTS54505
c2t5000CCA597D186A7d0 ONLINE 0 0 0 500.11 GB Hitachi HTS54505
c2t5000CCA597D18C3Bd0 ONLINE 0 0 0 500.11 GB Hitachi HTS54505
c2t5000CCA597D18EEDd0 ONLINE 0 0 0 500.11 GB Hitachi HTS54505
...spare-6 DEGRADED 0 0 0
.... c2t5000CCA597D192B5d0 FAULTED 0 0 0 too many errors 500.11 GB Hitachi HTS54505
.... c2t5000CCA597D1BC76d0 ONLINE 0 0 0 500.11 GB Hitachi HTS54505
c2t5000CCA597D036A8d0 ONLINE 0 0 0 500.11 GB Hitachi HTS54505
c2t5000CCA597D19759d0 ONLINE 0 0 0 500.11 GB Hitachi HTS54505
c2t5000CCA597D19C32d0 ONLINE 0 0 0 500.11 GB Hitachi HTS54505
...
spares
c2t5000CCA597D1BC76d0 INUSE currently in use 500.11 GB Hitachi HTS54505
Sooo, it appears that Napp-IT didn't remove the spare (I can't say I verified it actually removed it when i asked it to). The spare DID resilver when I did it manually, however it's just stuck there, showing degraded with the removed drive still FAULTED.
I'm open to suggestions at this point. I've tried everything I can think of (while off-site) to remove the issue short of reboot. I can't remove the spare now because it's INUSE. Clearing the status does nothing.
My next thought is going onsite, pulling the new drive, trying to remove the spare again, and putting it back in... if it doesn't change much I'll try to use the CLI to do another resilver. Failing that the only option I see is reboot and cross fingers.
Anyone else experienced this with an amicable outcome?
Last edited: