first faulty disk using ZFS

E

El_Kurgan

n00b
Joined
Dec 8, 2010
Messages
20
Hi all, after 2 weeks from the first boot, one of the OS disks has gone:

NAME STATE READ WRITE CKSUM
rpool DEGRADED 0 0 4
mirror-0 DEGRADED 0 0 8
c8d0s0 ONLINE 0 0 8
c8d1s0 FAULTED 1 242 0 too many errors

the question is: why can't the file be recovered? it's a mirror so I have to copies, or.. I have a faulty disk and the other has another kind of problem?

They are two identical ide 2,5 hard disks. (80gb) and 8gb ecc memory
 
What do you mean by "why can't the file be recovered?" What file? The above just shows that one drive of the mirror is toast - says nothing about not being able to recover a file - can you be more specific?
 
the cheksum errors has increased while I was doing a scrub and copying some files to other pool:

config:

NAME STATE READ WRITE CKSUM
rpool DEGRADED 0 0 7
mirror-0 DEGRADED 0 0 14
c8d0s0 ONLINE 0 0 14
c8d1s0 FAULTED 1 242 0 too many errors

errors: Permanent errors have been detected in the following files:

/export/home/kurgan/Descargas/aaaaaa.rar

this file can't be copyed, I get an error
 
Dude, you have major issues here. The 3rd number is the checksum error count, right? Note that the "good" drive has 14 errors listed, and the 2nd is outright dead. I think you'd better backup your pool asap and replace the drives :(
 
the checksums errors in the "good" drive could mean an error in the system memory or the ide controller?
 
The checksum error means that when zfs read the block back in from that disk, it failed checksum. Almost certainly a bad sector, failing controller, etc rather than system memory (I'd think you'd be seeing other things too if it were RAM, especially since you have ECC ram so you would need multibit errors to have corruption.)
 
Thanks for all the info.
I have set autoreplace on. How should I proceed? shutdown, put the new drive and the system will autoresilver automatically? or the disk must be detached first and use the replace command with the new drive? (both will be in the same port)
 
I think what I've seen recommended is to detach the old one, shut down, pull it, put the new one in, boot, and add it (don't remember if that resilvers automatically or not.)
 
1) check your memory with MemTest86+ first; never allow a storage system to touch your data without knowing the memory is not faulty.
2) reboot the system and make sure both devices are found; if they are your corrupted file may be recovered.

Such errors can be caused by working with defective RAM, or if your detached disk is failed and uBER is active on your remaining drives. But in that case you would see more READ errors; not checksum errors. Checksum errors mean CORRUPTION on the disk, or RAM caused the read chunk to fail checksum; so the RAM could also be at fault. Check that first.

Also check the SMART data from both of your disks, probably should post it here. This can be done with 'smartctl -A ...' command.
 
Not if the RAM is really faulty; you can see soon enough in MemTest86+ since it counts ECC errors too.

Checksum errors means something is corrupting; you need to quickly identify that issue. My experience:
1) memory errors
2) driver problems
3) disk errors (Samsung+NCQ without firmware upgrade)
4) some mysterious issue?
 
Well, I'm dubious. Any ram really *that* flaky would be crashing his box. I work on this stuff for a living (fault tolerant computers), and the multibit error rate for ECC ram is miniscule. Far likelier disk issues... maybe drives overheating, who knows...
 
I'll check memtest as soon I get back from work. I did a 24h test when I assembled the computer.

What is more strange for me is that I check de pools status almost every day, and the two faulty drivers has appeared at the same time.

It couldn't be a temperature problem. It doesn't have temperatura sensors, but the rest of my samsungs 204 are usually a 31º. I'll paste smart information as soon as i could.

thank you
 
I would check the following:

1) Swap all SATA cables. I've detected more defective cables than harddrives...
2) Check SMART data - 2.5" drives tend to have high load-cycle counts (fix: WDIDLE)
3) ECC ram can be deceptive - does your MB really support ECC (POST should identify as ECC)?

A detailed list of components would have been helpful here to draw more conclusions. It's not that common with 2x 2.5" drives in a server system so I am a bit curious (even if I use a 2.5" as boot drive myself).
 
Specs:

-gigabyte ga-ma770t-ud3
-amd x4 640
-2x4gb ecc 1333 kingstone
-2x80gb ide 2,5" 4200rpm (hitachi). mirroring v31
-6xsamsung 204 2tb. raidz2 v28
-lan intel gigabit pci-e 1x
-vga navidia quadro pci-e 1x
-power supply corsait 650w
-solaris express 11

The initial post don't says anything about ecc (I can't see the table that appears just before grub), but a new menu appears in the bios with options to configure ecc. Memtest detects ECC memory.

I can discard a temperature problem. At idle, the cpu (in bios) stays 2º above ambient temperature and hard disks 6 or 7.

Right now, I'm running memtest. By default memtest disables ecc test, should be enabled?
 
Well, I have replaced the hard disk and seems that all the problems are gone. It was impossible to "replace" and have to detach an attach the new one.

Only one more question: to resolve situattions like this.. what is the best way to resolve situations like this? There's any way to do a full image of the rpool and restore it in case that all disk fails?


thank you
 
If you have a big enough other disk, you could create a raid0 pool on it and do a 'zfs send' to it.
 
You must log in or register to reply here.
Back
Top