NFS vmdk's are NOT thin provisioned even though they should be. (ESX)

C

cyr0n_k0r

Supreme [H]ardness
Joined
Mar 30, 2001
Messages
5,360
Everywhere I read people tell me that the default behavior for vmdk's inside ESX when using NFS stores is to thin provision them.

Mine aren't for some reason. I've setup my NFS store, and I've already setup the vmkernel and added 1 100GB test data store however its showing up on the server hosting that share as 100GB even though there is no data inside of it yet.

Am I doing something wrong?
 
I've been looking at this actually myself - it looks like we may have changed that behavior abruptly - did you deploy from a template, by any chance, or convert from a physical machine? Or migrate to NFS from a VMFS datastore?
 
IIRC you can. Found some notes that I use:

SSH into the ESXi server and goto the directory with the new VM. Run this:
vmkfstools -c 20G -d thin -a lsilogic new-vm-name.vmdk

To expand a VMDK to 60GB you can run this command:
vmkfstools -X 60G placeholder.vmdk

If you copy a VM from ESX to ESXi the vmdk files will no longer be thin provisioned. To fix this you clone the vmdk on the target machine and replace the copied vmdk.

vmkfstools -i original.vmdk -d thin temp.vmdk
mv temp-flat.vmdk original-flat.vmdk
rm temp.vmdk
 
The VMDK is added as an additonal hard drive for one of our linux VM's. The vmdk is added to a windows NFS host share.

Did not deploy from a template or convert anything. Even the linux VM was setup from scratch (not clone or from template).

Berg0, your suggestion looks interesting. I am not running ESXi, will that make any difference?
How can I create the vmdk from the ESX host when its on an NFS store via command line? I want to setup an external NFS vmdk with a 900GB limit but THIN provisioned.
 
Berg0 said:
IIRC you can. Found some notes that I use:

SSH into the ESXi server and goto the directory with the new VM. Run this:
vmkfstools -c 20G -d thin -a lsilogic new-vm-name.vmdk

To expand a VMDK to 60GB you can run this command:
vmkfstools -X 60G placeholder.vmdk

If you copy a VM from ESX to ESXi the vmdk files will no longer be thin provisioned. To fix this you clone the vmdk on the target machine and replace the copied vmdk.

vmkfstools -i original.vmdk -d thin temp.vmdk
mv temp-flat.vmdk original-flat.vmdk
rm temp.vmdk
Click to expand...

You don't need to do this. For one thing, NFS is supposed to thin-provision from the start. Also, don't do the lsilogic portion - let it pick based on the defined OS of the VM (there's good reason for this) from the GUI if possible.

Second, NEVER USE MV/CP on a flat file vmdk. If block-size is different, you may not have a consistent file. Just don't get into the habit. Yes, it's safe on the same datastore, and on NFS, but just avoid getting into the habit. Trust me on this.
 
cyr0n_k0r said:
The VMDK is added as an additonal hard drive for one of our linux VM's. The vmdk is added to a windows NFS host share.

Did not deploy from a template or convert anything. Even the linux VM was setup from scratch (not clone or from template).

Berg0, your suggestion looks interesting. I am not running ESXi, will that make any difference?
How can I create the vmdk from the ESX host when its on an NFS store via command line? I want to setup an external NFS vmdk with a 900GB limit but THIN provisioned.
Click to expand...

Same commands. You shouldn't have to though.
 
lopoetve said:
Same commands. You shouldn't have to though.
Click to expand...

correct, I shouldn't have put the esxi part in there, vmkfstools works on ESX and ESXi. to be truthful before you do any of this on ESXi you need to actually get console access, which you are not supposed to have. I just copied whatever i had written down to remember the commands. I think I was using these for moving VM's between datastores without storage vmotion.

I'll take a quick look at work today and create a quick test VM on a few different datastores and see how it's configured by default. everything is automated now, so i haven't been manually creating VM's as of late.
 
ok, one thing -

ls in the console will show max allocated size, NOT currently allocated size.

vdf -h will show you the actual used space. Rely on it, NOT df or ls.
 
The NFS store holding the VMDK is a windows machine. So it is reporting to the NTFS filesystem that my test vmdk that I made 100GB is actually taking up all 100GB when in fact looking at it inside linux it's only using about 1.3 GB.
 
make a new windows VM, 5gb drive, and before installing anything see how big it is on the ntfs filesystem.
 
I have created a new windows VM with everything being default values. I gave the vm 8GB partition and stored it on the NFS. Before doing anything after that I checked the space used on the windows machine storing the vmdk and it reports a ******-flat.vmdk as 8GB in size.

It is not thin provisioning like it's supposed to.
 
cyr0n_k0r said:
I have created a new windows VM with everything being default values. I gave the vm 8GB partition and stored it on the NFS. Before doing anything after that I checked the space used on the windows machine storing the vmdk and it reports a ******-flat.vmdk as 8GB in size.

It is not thin provisioning like it's supposed to.
Click to expand...

I'll test against windows. It works on XFS and EXT3 shares - FWIW, Windows & NTFS are ~not~ supported storage, so it may be an oddity with that particular filesystem.

I'll test it tomorrow... gotta dig up another windows system.
 
cyr0n_k0r said:
Psh, in what world is a filesystem that is on ~90% of computers worldwide not supported?
Click to expand...

As a datastore for ESX? Since the beginning.

The only white-box NFS servers that are supported are 2 versions of RHEL, and 2 versions of Fedora.
 
Why is windows not a supported platform for NFS storage. That is stupid.
Have you been able to test on your own NTFS volume to see if you can replicate the issue?
 
cyr0n_k0r said:
Why is windows not a supported platform for NFS storage. That is stupid.
Have you been able to test on your own NTFS volume to see if you can replicate the issue?
Click to expand...

Because, most likely, it doesn't fully match the spec. Same reason the windows iSCSI target isn't supported (it returns an invalid page for something - I'd have to look it up again). And, additionally, we can't guarantee the performance/reliability of the hardware its on, or its suitability as a datastore.

I requisitioned the system - should get it monday to test. I have to nab 2 machines for this kind of test, since we don't have a windows NFS server lying around :)
 
all right. Initial tests seem to confirm - it looks like we're using fully-allocated VMDKs all right. Interesting :)

I'm mounting the drive to another machine to see if it just "thinks" it's using that, or if it actually is.

I don't know the reasoning yet. I did confirm that we do thin-provision on RHEL.

edit: Yep. doesn't work.
edit2: fwiw, it seems like windows NFS is extremely slow - as in, usable for ISO/template storage only. Definitely not for running virtual machines. I'd build a RHEL box. Or FreeNAS.
 
Last edited:
lopoetve said:
Second, NEVER USE MV/CP on a flat file vmdk. If block-size is different, you may not have a consistent file. Just don't get into the habit. Yes, it's safe on the same datastore, and on NFS, but just avoid getting into the habit. Trust me on this.
Click to expand...

So using SCP to download files from a datastore (so I can recreate the datastore with a new block size) is a bad thing? Or is this ok?
 
SpaceHonkey said:
So using SCP to download files from a datastore (so I can recreate the datastore with a new block size) is a bad thing? Or is this ok?
Click to expand...

SSH works :) Don't know the details of why per say, but we've seen the mv/cp fail and never seen scp fail.

Then again, I also haven't seen mv/cp fail since the 3.0.1 era... it may have been a bug with that, for all I know - I just keep repeating that mantra, to be honest. I was always taught to avoid it as much as possible, and that it wasn't safe... there are better ways, to put it simply, for most things.
 
here's something interesting...

NFSoutput.JPG


That's a 4gb VM on a RHEL NFS mount.

LS shows 4gb. Du shows use of only 186k (Which matches, since I haven't even booted the vm yet). I'm wondering if windows just can't interpret it correctly?
 
I have a program called "dirsize" that recurses our directories to see how large they are. When it goes through the folder where the vmdk is located on the windows machine it reports only 4KB usage in that directory. It's like it doesn't even see the vmdk file.

I also did a test where I added about 2GB to the 100GB test vmdk and dirsize still reported only 4KB being used. So it looks like it doesnt report the size correctly either way.
 
cyr0n_k0r said:
I have a program called "dirsize" that recurses our directories to see how large they are. When it goes through the folder where the vmdk is located on the windows machine it reports only 4KB usage in that directory. It's like it doesn't even see the vmdk file.

I also did a test where I added about 2GB to the 100GB test vmdk and dirsize still reported only 4KB being used. So it looks like it doesnt report the size correctly either way.
Click to expand...

Windows might have some kind of delayed write... try rebooting the server after adding 2gb to it and see what happens, just in case - that'll make it flush the cache.

This is mighty strange, tbh - and it's not a well documented feature either way.
 
You must log in or register to reply here.
Back
Top