ESXi chkdsk equivalent?

[Zarathustra[H]]

Hey all,

Had an odd issue on my ESXi 5.1 server yesterday.

I had 4 guests running (Ubuntu 12.04 headless server, pfSense, Windows XP and Windows Vista.)

Suddenly performance became terrible. The client became mostly unresponsive, but would respond in fits and starts. I was able to reboot my Ubuntu guest, and upon boot it was complaining about read errors from /sda, which is just a standard vmware image file on my datastore. (an SSD where all my guest images are stored)

Uh oh, I thought, I must have a failing drive. At this point I had lost the ability to control the server via the Vsphere client and decided to reboot the server by either plugging in a keyboard locally and forcing a restart or just cutting power to it, and starting it back up again.

When the server started back up again, everything worked like normal again. There were no signs of the drive read issues I had had prior to the reboot. All of my guests, except my Vista guest, booted back up normally again.

My Vista guest on the other hand was borked.

The Vista boot loaded could not find an OS to boot, and booting from the Vista CD to do a repair did not help either. It's like the Vista partition inside the Vmware image file just decided to empty itself.

So, not sure what caused all this, but I suspect a drive problem.

Is there a good way to run some sort of disk diagnostic on datastores within ESXi? How do I go about doing that?

Thanks,
Matt

 

[Hagrid]

Can you boot a linux cd and run a disk utility from it?

 

[Zarathustra[H]]

Can you boot a linux cd and run a disk utility from it?

I could.

What file system does ESXi format it's local drugstores in?

 

[dasaint]

the Filesystem is VMFS, and as far as i am aware there are no such scandisk type tools...

as a backup i would consider migrating the VM data ASAP and then doing HDD Tests with the Company Software you could use ESXtop to see if the drive is acting abnormally but i would also caution that if the drive is going bad and is causing data corruption the first key is to get those vms safe asap...

 

[Eulogy]

Are you certain you were not over-committed in resources (i.e. have 10GB allocated to VMs and being used, where as host only has 8GB as an example)? What do your log files show you?

 

[lopoetve]

5.1?

Run:
voma -m vmfs -f check -d /vmfs/volumes/NAA_OF_YOUR_DATASTORE:YOUR_PARTITION_NUMBER

Paste output here.

esxcfg-scsidevs -m will identify the naa -> datastorename mappings.

 

[Zarathustra[H]]

5.1?

Run:
voma -m vmfs -f check -d /vmfs/volumes/NAA_OF_YOUR_DATASTORE:YOUR_PARTITION_NUMBER

Paste output here.

esxcfg-scsidevs -m will identify the naa -> datastorename mappings.

Thank you.

I'll try this.

While I am rather experienced in unix-like consoles, ESXi seems to have changed a lot, and I haven't had the opportunity to play around in there yet.

 

[dasaint]

Oh LOP go enjoy Barcelona!!! TRADE???

 

[Zarathustra[H]]

5.1?

Run:
voma -m vmfs -f check -d /vmfs/volumes/NAA_OF_YOUR_DATASTORE:YOUR_PARTITION_NUMBER

Paste output here.

esxcfg-scsidevs -m will identify the naa -> datastorename mappings.

Question,

Do I have to enter Maintenance mode, or shut down my guests on the drive in order to do this, or can I run it with everything on?

 

[dasaint]

VOMA – vSphere On-disk Metadata Analyzer

VOMA is a new customer facing metadata consistency checker tool, which is run from the CLI of ESXi 5.1 hosts. It checks both the Logical Volume Manager (LVM) and VMFS for issues. It works on both VMFS-3 & VMFS-5 datastores. It runs in a check-only (read-only) mode and will not change any of the metadata. There are a number of very important guidelines around using the tool. For instance, VMFS volumes must not have any running VMs if you want to run VOMA. VOMA will check for this and will report back if there are any local and/or remote running VMs. The VMFS volumes can be mounted or unmounted when you run VOMA, but you should not analyze the VMFS volume if it is in use by other hosts.

If you find yourself in the unfortunately position that you suspect that you may have data corruption on your VMFS volume, prepare to do a restore from backup, or look to engage with a 3rd party data recovery organization if you do not have backups. VMware support will be able to help in diagnosing the severity of any suspected corruption issues, but they are under no obligation to recover your data.

I’m sure you will agree that this is indeed a very nice tool to have at your disposal.

See http://cormachogan.com/2012/09/04/vsphere-5-1-storage-enhancements-part-1-vmfs-5/

 

[lopoetve]

Question,

Do I have to enter Maintenance mode, or shut down my guests on the drive in order to do this, or can I run it with everything on?

All guests shut down > not. You'll get a LOT of spurious messages if they're up and running, but I know to discount those.

 

[lopoetve]

VOMA – vSphere On-disk Metadata Analyzer

VOMA is a new customer facing metadata consistency checker tool, which is run from the CLI of ESXi 5.1 hosts. It checks both the Logical Volume Manager (LVM) and VMFS for issues. It works on both VMFS-3 & VMFS-5 datastores. It runs in a check-only (read-only) mode and will not change any of the metadata. There are a number of very important guidelines around using the tool. For instance, VMFS volumes must not have any running VMs if you want to run VOMA. VOMA will check for this and will report back if there are any local and/or remote running VMs. The VMFS volumes can be mounted or unmounted when you run VOMA, but you should not analyze the VMFS volume if it is in use by other hosts.

If you find yourself in the unfortunately position that you suspect that you may have data corruption on your VMFS volume, prepare to do a restore from backup, or look to engage with a 3rd party data recovery organization if you do not have backups. VMware support will be able to help in diagnosing the severity of any suspected corruption issues, but they are under no obligation to recover your data.

I’m sure you will agree that this is indeed a very nice tool to have at your disposal.

See http://cormachogan.com/2012/09/04/vsphere-5-1-storage-enhancements-part-1-vmfs-5/

Forgot - we added the flag to only allow if quiesced.

 

[Zarathustra[H]]

Awesome.


Didn't even need to look up NAA's.

ESXi was kind enough to add a symlink with the configured datastore name in /vmfs/volumes pointing to it.

I am getting an error message when running this command though:

 # voma -m vmfs -f check -d /vmfs/volumes/505d81bc-9505cd49-2010-6805ca018ab0
Checking if device is actively used by other hosts
         ERROR: Failed to check for heartbeating hosts on device'/vmfs/volumes/505d81bc-9505cd49-2010-6805ca018ab0'

Advice?

I have two drives in the system, and get the same error on both...

Thanks,
Matt

 

[Zarathustra[H]]

Awesome.


Didn't even need to look up NAA's.

ESXi was kind enough to add a symlink with the configured datastore name in /vmfs/volumes pointing to it.

I am getting an error message when running this command though:

 # voma -m vmfs -f check -d /vmfs/volumes/505d81bc-9505cd49-2010-6805ca018ab0
Checking if device is actively used by other hosts
         ERROR: Failed to check for heartbeating hosts on device'/vmfs/volumes/505d81bc-9505cd49-2010-6805ca018ab0'

Advice?

I have two drives in the system, and get the same error on both...

Thanks,
Matt

FWIW, Rebooting the server or entering maintenance mode does not appear to ahve any effect.

I get the same error message no matter what.

 

[lopoetve]

Any other hosts talking to that lun?

edit:
Oh, you're running it on the uuid. You have to run it on /vmfs/devices/disks/naa:1