ZFS Data Corruption

[x_rufus_x]

Hello Everyone.

I'm experiencing some problems with my Open Indiana ZFS Server.

One of my pools is showing some data corruption. After a zpool status I had 27 CKSUM errors, 13 READ Errors and a list of 8 files with permanent errors, my pool is already on a DEGRADED State.

I runned a scrub and after some hours It was unable to repair any of the files. Those corrupted files are not important, and I can get rid of them.

So, What is the best way to clean those files from my pool and try to put it on ONLINE State?

Thanks,
In advance

ps: I'm not an expert in Solaris/ZFS

ps2: Sorry for my bad english

 

[brutalizer]

Why do you have corrupted data? What happened? Is your hardware faulty? If ZFS can not repair the data, it is some serious problems. You should investigate the problems.

 

[x_rufus_x]

Why do you have corrupted data? What happened? Is your hardware faulty? If ZFS can not repair the data, it is some serious problems. You should investigate the problems.

The ZFS file server is a VM on a ESX Host and I had some problems this week with this host .
All my VMs crashed. Some VMs could not even start, and those that could start, after some minutes restarted.
It seems It was the dust that made my PSU not work properly. I cleanned the machine today and since then it worked as usual. (10h strait)

After that issue with my ESX, this errors began to pop up on my ZFS. Maybe my HD is damaged.

I have other two pools, and They'r working fine.

 

[QQ~]

Are the zpools raidz, basic, mirror?

 

[bAMtan2]

your english is good.

but I don't think dust causes zfs data corruption. you have some serious work to do if you want to ensure your data is safe, and I'm afraid you might not know what to do...

 

[TType85]

you have some serious work to do if you want to ensure your data is safe, and I'm afraid you might not know what to do...

This is why I like using an off the shelf NAS's that have at least some support over the OI/FreeNas/Etc roll your own setups. They are easy to set up but when stuff happens...

I use FreeNas myself for one of my servers and had a drive going bad and had errors. I am not sure in OI how you can list the drives to see if it's one going bad. IIRC I was able to swap the drive in and run a command (not a expert here either) and it was back up and running healthy.

 

[x_rufus_x]

Are the zpools raidz, basic, mirror?

The one that has problems It's basic (It's a single 2TB disk)

your english is good.

but I don't think dust causes zfs data corruption. you have some serious work to do if you want to ensure your data is safe, and I'm afraid you might not know what to do...

I really don't know what to do . Reading this page about the error (http://illumos.org/msg/ZFS-8000-8A), It says that maybe I have to re build the entire pool

 

[_Gea]

ZFS offers two fundamental features:

End to end checksums on all data:
- this feature reports any error or corruption on your data. Errors from single bit errors up to disk failures are common on disks.

Self healing:
When ZFS detects errors it can repair these errors BUT ONLY if you offer raid redundancy. Without, your data is lost like with any other filesystem. You then need a backup.

What you should do:
Add a second 2 TB disk to your basic disk to build a mirror.

 

[staticlag]

You can clear the errors and the degraded state. IE. "reset" the drive by using:

sudo zpool clear <poolname> <drive>

If you have only 1 drive a scrub will do nothing, it will only report the errors but not fix them. As said before you need to have either a mirror or parity (raidz, raidz2) configuration to be able to correct errors like this.

Example:

  pool: rpool
 state: ONLINE
  scan: scrub repaired 0 in 0h1m with 0 errors on Sat Jun  1 04:01:01 2013
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  DEGRADED      0     0     0
            c7t1d0  DEGRADED      0     56     0
            c7t3d0  ONLINE       0     0     0

If i want to reset drive c7t1d0

I would do:

sudo zpool clear rpool c7t1d0

 

[Billy_nnn]

Issues like this can be caused by the disk, controller, psu and/or cabling.

If your other pools are functioning OK, then there's a good probability that the PSU is OK (though you can confirm this when you switch the cabling round)

Start at the basics, and do one thing at a time only

Switch the cabling with that from a known good drive - then run a scrub on the failed pool.
(if the errors are gone, run a scrub on the pool which now has the original cabling)

If the errors persist on the same drive, export all pools and shutdown.
Switch the port that the failing drive is connected to, with a known good sata port (from one of your other zfs data drives). Reboot and re-import all pools - then run a scrub on the failed pool.
(if the errors are gone, run a scrub on the pool which now has the disk connected to the original failing port)

If the errors still persist on the same drive, you are likely looking at a disk problem.
You can just delete the affected files and run a zpool clear and then a zpool scrub - that should clear the errors (assuming no new ones are found). However you then need to deal with the underlying disk problem.
Run a whole disk scan several times and see if the same sectors are being reported every time - if so it looks like media damage - if they are random you may be looking at a controller or head problem.
You can use "format-analyze-read" (ensure no write tests are selected) to run the scan - observe the output and check in /var/adm/messages for any disk related errors.

Replacement is fairly easy - as Gea said earlier, just attach a new drive and add into the pool with the failing disk, as a mirror - when the resilver is complete, detach the failed drive from the mirror, leaving just the new drive in the pool.

 

[devman]

Just to clear up minor point. You can protect yourself from data degradation with a single disk vdev by setting the zfs attribute "copies" to a value greater than 1. That will allow scrub to repair on single disk vdevs. Obviously this does not protect you if the drive fails completely though.

The "copies" attribute only affects future writes and not existing data on the filesystem, so you'd have to rewrite everything already on the disk to have it protected.

 

[brutalizer]

The one that has problems It's basic (It's a single 2TB disk)

So you used ZFS on a single disk, without any redundancy? No mirror, no raid, no nothing? In that case it explains why ZFS can not repair those files. If you had used mirror, ZFS would have repaired those files.

I suggest you use some sort of redundancy in the future, such as mirror or raid. Then your data will be safer.

 

[Mini-Me]

ZFS offers two fundamental features:

End to end checksums on all data:
- this feature reports any error or corruption on your data. Errors from single bit errors up to disk failures are common on disks.

Self healing:
When ZFS detects errors it can repair these errors BUT ONLY if you offer raid redundancy. Without, your data is lost like with any other filesystem. You then need a backup.

What you should do:
Add a second 2 TB disk to your basic disk to build a mirror.

I'm not an expert on ZFS, and in fact I haven't even used it, but I know there are some caveats with adding a new disk to an existing RAID-Z array, so I want to ask this right now in case there are any similar caveats in the case of x_rufus_x going from one disk to a mirror:
If you simply add a new disk to an existing zpool, does ZFS do the "wrong thing" and make the disk a single point of failure for the entire array? Or is this just an issue with RAID-Z?
References for the RAID-Z issue:
http://forums.freenas.org/threads/raidz-wont-rebuild-on-single-drive-fail.11759/#post-54007
http://forums.freenas.org/threads/adding-disk-to-raid.8730/

 

[devman]

I'm not an expert on ZFS, and in fact I haven't even used it, but I know there are some caveats with adding a new disk to an existing RAID-Z array, so I want to ask this right now in case there are any similar caveats in the case of x_rufus_x going from one disk to a mirror:
If you simply add a new disk to an existing zpool, does ZFS do the "wrong thing" and make the disk a single point of failure for the entire array? Or is this just an issue with RAID-Z?
References for the RAID-Z issue:
http://forums.freenas.org/threads/raidz-wont-rebuild-on-single-drive-fail.11759/#post-54007
http://forums.freenas.org/threads/adding-disk-to-raid.8730/

You can only 'zpool add' vdevs If you specify only a single block device as an argument it will add that as a vdev and yes it will become a single point of failure. If you try to add a vdev that reduces the redundancy factor of the pool I believe you get a warning and confirmation before it actually adds the vdev.

You can not add or remove members from a vdev, you can only replace members or attach and detach mirrors. To answer your question directly, you cannot add disks to a raidz.

 

[danswartz]

You can't add a disk to a raid-z* vdev. In theory you can add a simple disk to an existing pool (with a raid-z* vdev) without protection, but bad idea. AFAIK, it should warn you and require a '-f' (force) switch on the zpool add command. Mirroring an existing single-disk pool is trivial. 'zpool attach POOL OLD-DISK NEW-DISK' and done...

 

[Mini-Me]

You can only 'zpool add' vdevs If you specify only a single block device as an argument it will add that as a vdev and yes it will become a single point of failure. If you try to add a vdev that reduces the redundancy factor of the pool I believe you get a warning and confirmation before it actually adds the vdev.

You can not add or remove members from a vdev, you can only replace members or attach and detach mirrors. To answer your question directly, you cannot add disks to a raidz.

So you CAN attach a mirror? That's good. I knew you couldn't add a disk to RAID-Z without backing up your data first and creating a totally new array, but I just wanted to make sure x_rufus_x wouldn't get similarly bitten trying to create a mirror.

 

[danswartz]

The wording was not clear. You can attach/detach members of a mirror - you cannot detach a mirrored vdev from the pool. You can attach any kind of vdev to an existing pool, not just mirrors. I think maybe what was meant was: "you can attach TO a mirror"?

 

[Mini-Me]

I'm not familiar enough with all the terminology to know the implications of the words I'm speaking (lol), because I'm not sure if the two "parts" of the mirror count as a single vdev or two. However, I'll do my best to rephrase:
x_rufus_x currently has his zpool set up with a vdev containing a single disk. If he wants to add a disk to create a mirror setup, could he safely add it to his current setup, or would he have to create a new zpool entirely? That is, is a single-disk setup convertible into a mirrored setup without starting from scratch?

 

[devman]

That is, is a single-disk setup convertible into a mirrored setup without starting from scratch?

Yes. You can attach a new mirror to a vdev member at any time.

 

[x_rufus_x]

Yes. You can attach a new mirror to a vdev member at any time.

That's cool!

Unfortunatelly, my problem is way worse than I thought. After almost one day working ok, my ESXi became inacessible again (and all my VMS, including my ZFS FileServer).

So, until I find the real problem, I think I'll install an OpenIndiana on a Old HD, migrate the pools (I think that is possible ) to it and backup my files.

So,
Thanks for all messages

 

[danswartz]

A vdev is one or more disks as either solo or a raid level. e.g. 3 disks in a raidz1 is a vdev. you can then add 3 more disks as another raidz1 and that is 'striped' with the first one. or 2 2-disk mirrors striped (e.g. raid10).