Vsphere 5.5 Corrupt Datastore Help?

[Zarathustra[H]]

Hey all,

I'm trying to figure out how to scan and attempt to repair a datastore in ESXi after a recent power outage (I really ought to invest in a UPS), and would appreciate some help!


First some details (if they are of any relevance):

When bringing the server back up after the power outage I was faced with this error message (all other VM's booted up just fine):

At first I thought it was just the vmdk that was corrupted. I checked my backup folder and found that I had forgotten to back up this VM, so I decided to reinstall. Just in case anything could be saved I decided to copy all the VM files to my backup drive before reinstalling. That is when I got this message:

I had originally thought it was the filesystem inside the vdmk that was corrupt, but since I can't even copy the file, I realize it is the datastore filesystem itself that is corrupted.

I realize I will probably have to reinstall the VM one way or another, but I need to make sure the datastore filesystem is repaired before I proceed. Problem is, I can't find a way to do this.

I have been digging around the menus in the client for a file system scanner or something like that, but can't find one. I did some googling, but since VmWare has its own terminology for everything, I think I haven't been pulling up the right guides.

This is the drive I am having issues with (particularly the 24.98GB VMFS partition):

Any suggestions on how to proceed would be appreciated!

Thank you,
Matt

 

[Zarathustra[H]]

Hmm. Been playing around with partedUtil a little bity, but I can't seem to find a built in scan tool... Any help would be appreciated.

 

[Zarathustra[H]]

So, I found this guide on how to scan a partition with voma.

I shut down all VM's, and put the host in maintenance mode.

Then I initialized a scan using the following command line:

~ # voma -m vmfs -f check -d /vmfs/devices/disks/t10.ATA_____OCZ2DONYX________________________________LR8ACJC71WEYV3A34GR7:3
Checking if device is actively used by other hosts           
Found 1 actively heartbeating hosts on device '/vmfs/devices/disks/t10.ATA_____OCZ2DONYX________________________________LR8ACJC71WEYV3A34GR7:3'
1): MAC address 90:2b:34:69:12:31

I'm not quite sure what a heartbeat is in Vsphere language, and how I might disable it in order to scan the disk.

What's more concerning though is that the guide says this:

The tool can only find errors – but not fix them. So if VOMA detects any errors, please consult VMware support for further help.

Is there seriously no way to find and repair corrupt partitions without contacting support? That seems.... Well... Lacking?

I'm on a free license for my home use. If I can't find a way to scan and repair the VMFS partition, I am going to have to reformat it and start over, (after backing up my remaining VM:s of course) but I'm not sure if I can do that with this heartbeating thing going on.

Would really appreciate any help here!

Thanks,
Matt

 

[Zarathustra[H]]

Hmm..


I was thinking I could maybe boot from a Linux rescue disk and scan it using parted, but parted seems to lack VMFS support as well...

Is there no open source vmfstools package or anything like that?

 

[Zarathustra[H]]

Actually, I takje that back. There is a vmfs-tools package in Ubuntu, and it seems to include an fsck.vmfs binary...

I'm going to have to try this...

 

[Zarathustra[H]]

Hmm...

The man page mentions integrity checks, but nothing about fixing...

FSCK.VMFS(8)                                                            vmfs-tools                                                           FSCK.VMFS(8)

NAME
       fsck.vmfs - VMFS file system check utility

SYNOPSIS
       fsck.vmfs VOLUME...

DESCRIPTION
       The fsck.vmfs performs various integrity checks on a VMFS file system.

       The VOLUME to be opened can be either a block device or an image file. When the VMFS spreads accross several extents, all extents must be given.

AUTHORS
       Christophe Fillot <cf@utc.fr[1]>, Mike Hommey <mh@glandium.org[2]>

SEE ALSO
       debugvmfs(8)

NOTES
        1. cf@utc.fr
           mailto:cf@utc.fr

        2. mh@glandium.org
           mailto:mh@glandium.org

Has anyone tried this? Can it actually repair partitions, or just tell you if they are broken (which I already know it is...)

 

[nicholasfarmer]

The only way I know to fix corruption is to DD over the corrupted bits. I've sent a dd of my vol to VMware support and they fixed it, (zero out the corruption) then you can dd the fix over the vol.

Sucks but it gives you enough of a fix to migrate the VM off of the lun so you can destroy and format it.

If you have a backup just format and restore! You are lucky it's 23GB and not 4TB or greater..

Nicholas Farmer

 

[Zarathustra[H]]

The only way I know to fix corruption is to DD over the corrupted bits. I've sent a dd of my vol to VMware support and they fixed it, (zero out the corruption) then you can dd the fix over the vol.

Sucks but it gives you enough of a fix to migrate the VM off of the lun so you can destroy and format it.

If you have a backup just format and restore! You are lucky it's 23GB and not 4TB or greater..

Nicholas Farmer

Thank you!

Two questions:

1.) Will formatting the partition with the heartbeat still on cause problems? If so how can I shut it off?

2.) Can I format it with a different file system (like ext4) so I don't have this problem again?

Thanks,
Matt

 

[lopoetve]

Actually, I takje that back. There is a vmfs-tools package in Ubuntu, and it seems to include an fsck.vmfs binary...

I'm going to have to try this...

STOP

Those utilities were designed for VMFS3, not 5.

Heartbeat - a host is accessing that datastore, which is sane, since it's, well, a datastore.

How do you know it's corrupt - a missing file could do this, vs being corrupt. You need to do the following:

1. From the CLI, run: vmkfstools -V; sleep 2; cat /var/log/vmkernel.log | grep -i 'invalid|error|missing'
2. post the output.

We'll go from there. I'll get you fixed if I can.

 

[lopoetve]

Thank you!

Two questions:

1.) Will formatting the partition with the heartbeat still on cause problems? If so how can I shut it off?

2.) Can I format it with a different file system (like ext4) so I don't have this problem again?

Thanks,
Matt

No - Given how our virtual machines work, it needs VMFS for shared access/etc. Don't worry, we'll get it fixed.

 

[lopoetve]

So, I found this guide on how to scan a partition with voma.

I shut down all VM's, and put the host in maintenance mode.

Then I initialized a scan using the following command line:

~ # voma -m vmfs -f check -d /vmfs/devices/disks/t10.ATA_____OCZ2DONYX________________________________LR8ACJC71WEYV3A34GR7:3
Checking if device is actively used by other hosts           
Found 1 actively heartbeating hosts on device '/vmfs/devices/disks/t10.ATA_____OCZ2DONYX________________________________LR8ACJC71WEYV3A34GR7:3'
1): MAC address 90:2b:34:69:12:31

I'm not quite sure what a heartbeat is in Vsphere language, and how I might disable it in order to scan the disk.

What's more concerning though is that the guide says this:



Is there seriously no way to find and repair corrupt partitions without contacting support? That seems.... Well... Lacking?

I'm on a free license for my home use. If I can't find a way to scan and repair the VMFS partition, I am going to have to reformat it and start over, (after backing up my remaining VM:s of course) but I'm not sure if I can do that with this heartbeating thing going on.

Would really appreciate any help here!

Thanks,
Matt

You have to remember - fixing a bad volume is normally something the journal does. If it can't, that means you have something where it doesn't know what is "right", just that it expected to find a value and instead found an elephant wearing a fancy hat, which is obviously "wrong".

The fix portion is locked out because all it can do to "fix" it is put back what would be there on a blank volume - that means data goes missing. Since that is also normally called "bad" to most people, it only lets you detect what is going on, so that support can find a way to "fix" it, or at least tell you what has gone wrong, since humans can interpret elephants with fancy hats.

That's what I'm here for, in other words. So get me that data, and we'll see what kind of elephant you have!

 

[Zarathustra[H]]

STOP

Those utilities were designed for VMFS3, not 5.

Heartbeat - a host is accessing that datastore, which is sane, since it's, well, a datastore.

How do you know it's corrupt - a missing file could do this, vs being corrupt. You need to do the following:

1. From the CLI, run: vmkfstools -V; sleep 2; cat /var/log/vmkernel.log | grep -i 'invalid|error|missing'
2. post the output.

We'll go from there. I'll get you fixed if I can.

Thank you very much for the warning, and offer of assistance. I will post back with this info ASAP.

 

[Zarathustra[H]]

No - Given how our virtual machines work, it needs VMFS for shared access/etc. Don't worry, we'll get it fixed.

Ahh, that is too bad. I use mine in a standalone setup without any migration or anything like that. Any exceptions there? I mean, the Workstation version can run multiple vdmk's simultaneously off of my Mint installs ext4 partitions...

You have to remember - fixing a bad volume is normally something the journal does. If it can't, that means you have something where it doesn't know what is "right", just that it expected to find a value and instead found an elephant wearing a fancy hat, which is obviously "wrong".

The fix portion is locked out because all it can do to "fix" it is put back what would be there on a blank volume - that means data goes missing. Since that is also normally called "bad" to most people, it only lets you detect what is going on, so that support can find a way to "fix" it, or at least tell you what has gone wrong, since humans can interpret elephants with fancy hats.

That's what I'm here for, in other words. So get me that data, and we'll see what kind of elephant you have!

Yeah, in my case I've already resigned myself to the fact that my FreeNAS boot image is unrecoverable. I just want to start over, but I want to make sure the datastore isn't corrupt before I do.

 

[Zarathustra[H]]

STOP

Those utilities were designed for VMFS3, not 5.

Heartbeat - a host is accessing that datastore, which is sane, since it's, well, a datastore.

How do you know it's corrupt - a missing file could do this, vs being corrupt. You need to do the following:

1. From the CLI, run: vmkfstools -V; sleep 2; cat /var/log/vmkernel.log | grep -i 'invalid|error|missing'
2. post the output.

We'll go from there. I'll get you fixed if I can.

So, I tried this. I didn't get any results. Is there anything else I should be running grepping for?

I tried instead grepping for not found (as that was the error message in the client gui) and came up with this:

2014-04-11T18:44:37.498Z cpu3:33925)WARNING: Fil3: 15068: Found invalid object on 505d6f72-5e4b31b1-ce95-6805ca018ab0 <FD c3392 r293> expected <FD c11 r11>
2014-04-11T18:44:37.498Z cpu3:33925)Vol3: 2612: Failed to get object 28 type 3 uuid 505d6f72-5e4b31b1-ce95-6805ca018ab0 FD 2c002c4 gen 15 :Not found
2014-04-11T18:44:37.499Z cpu3:33925)WARNING: Fil3: 15068: Found invalid object on 505d6f72-5e4b31b1-ce95-6805ca018ab0 <FD c3392 r293> expected <FD c11 r11>
2014-04-11T18:44:37.499Z cpu3:33925)Vol3: 2612: Failed to get object 28 type 3 uuid 505d6f72-5e4b31b1-ce95-6805ca018ab0 FD 2c002c4 gen 15 :Not found

Does this help us at all? It seems to be referring to the correct UUID

I also tried grepping for the UUID, and only got the same 4 lines.

Thanks,
Matt

 

[lopoetve]

Yeah, that's a problem. There's an elephant where your directory entry is supposed to be.

My grep should have caught that on the invalid...

The pointers for the direntry are way into la-la land - like off the end of the disk la-la land. That might be fixable, but probably not.

Do:

dd if=vmfs/devices/disks/t10.ATA_____OCZ2DONYX________________________________LR8ACJC71WEYV3A34GR7:3 of=/tmp/harddump.bin bs=1M count=1500
Gzip the harddump.bin, find a place to stick it where I can get it (should be about 300-400M).

 

[Zarathustra[H]]

Thank you, I really do appreciate the help!

I will run the DD, and will ahve to figure out where to put it. Normally that would have been on my FreeNAS guest with sftp access, but well, it is down

I appreciate that you are going through all the trouble, but wouldn't it be just as easy to just copy my "good" vm's off the partition, reformat it, copy them back, and reinstall the broken one? I really haven't lost anything at this point, except the boot partition for my FreeNAS guest, and it doesn't take very long to reinstall and set back up. Trying to repair a dd of a partition sounds to me like it would be more work, and I would be putting that on you (as much as I do appreciate the help!)

If I were to go down the reformat road, do I have to do anything about the heartbeat first?

Meanwhile I will start thinking about where I can share a 300M+ file.

Thanks,
Matt

 

[Zarathustra[H]]

Hmm..

Looks like it's not accepting the dd syntax, and it's not giving any detailed error message,,,,

Maybe I am just very tired, but I can't seem to figure out what's wrong right now, it looks correct to me.

Odd..

This is what I get:

~ # dd if=vmfs/devices/disks/t10.ATA_____OCZ2DONYX________________________________LR8ACJC 71WEYV3A34GR7:3 of=/tmp/harddump.bin bs=1M count=1500
BusyBox v1.20.2 (2012-12-11 11:54:28 PST) multi-call binary.

Usage: dd [if=FILE] [of=FILE] [ibs=N] [obs=N] [bs=N] [count=N] [skip=N]
	[seek=N] [conv=notrunc|noerror|sync|fsync]

Copy a file with converting and formatting

	if=FILE		Read from FILE instead of stdin
	of=FILE		Write to FILE instead of stdout
	bs=N		Read and write N bytes at a time
	ibs=N		Read N bytes at a time
	obs=N		Write N bytes at a time
	count=N		Copy only N input blocks
	skip=N		Skip N input blocks
	seek=N		Skip N output blocks
	conv=notrunc	Don't truncate output file
	conv=noerror	Continue after read errors
	conv=sync	Pad blocks with zeros
	conv=fsync	Physically write data out before finishing

Numbers may be suffixed by c (x1), w (x2), b (x512), kD (x1000), k (x1024),
MD (x1000000), M (x1048576), GD (x1000000000) or G (x1073741824)

So it's just displaying the --help screen without any details...

I will have to take a fresh look at this tomorrow morning.

Thanks again for all the help.

--Matt

 

[gantux]

Try again with the space character removed between ...__LR8ACJC
and 71WEYV3A34GR7:3

 

[lopoetve]

Yeah, that's Hardforum doing goofy things to the formatting. Tab complete works too.

 

[lopoetve]

Thank you, I really do appreciate the help!

I will run the DD, and will ahve to figure out where to put it. Normally that would have been on my FreeNAS guest with sftp access, but well, it is down

I appreciate that you are going through all the trouble, but wouldn't it be just as easy to just copy my "good" vm's off the partition, reformat it, copy them back, and reinstall the broken one? I really haven't lost anything at this point, except the boot partition for my FreeNAS guest, and it doesn't take very long to reinstall and set back up. Trying to repair a dd of a partition sounds to me like it would be more work, and I would be putting that on you (as much as I do appreciate the help!)

If I were to go down the reformat road, do I have to do anything about the heartbeat first?

Meanwhile I will start thinking about where I can share a 300M+ file.

Thanks,
Matt

I'm a sucker for "fix shit" if you can. Reformat is too much the windows way, although we may still end up there.

 

[Zarathustra[H]]

Try again with the space character removed between ...__LR8ACJC
and 71WEYV3A34GR7:3

Ahh yes. I must have been WAAY too cross-eyed to see that the other night.

I'm a sucker for "fix shit" if you can. Reformat is too much the windows way, although we may still end up there.

Please check your PM for a link to the file.

Thank you again!

--Matt