ZFS free space missing after deletes

[wheelz]

In my testing of switching to a ZFS box, I have come across a seemingly simple problem that I thought it would be good to start a new thread. My testing thread is here for my config, etc:

http://hardforum.com/showthread.php?t=1608982

During my testing I have been copying and deleting VMs from my zvol (vmfs formatted with compress and dedup on). I have copied several hundred GBs of VMs to it and now I have completely deleted all VMs off the volume. There are no snapshots and since all files have been deleted, even dedup'd data should be freed up as there are no references at all (since all files are gone). However here is what zpool list shows me:

NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
dpool01 4.53T 622G 3.92T 13% 1.02x ONLINE -

zfs list:

NAME USED AVAIL REFER MOUNTPOINT
dpool01 512G 3.08T 55.9K /dpool01
dpool01/vmfs_vols 508G 3.08T 54.3K /dpool01/vmfs_vols
dpool01/vmfs_vols/GenVM01 508G 3.08T 508G -

zfs list -o space:

NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
dpool01 3.08T 512G 0 55.9K 0 512G
dpool01/vmfs_vols 3.08T 508G 0 54.3K 0 508G
dpool01/vmfs_vols/GenVM01 3.08T 508G 0 508G 0 0

Does anyone know why this is or how to get zfs to "see" it's truly available space?

 

[danswartz]

I think this is normal. The reason is that a zvol is an opaque object to zfs - it has no idea what is going on internally, so as vmfs objects are created and deleted, the zvol will eventually "fill up", since zfs is a copy on write filesystem.

 

[wheelz]

My other option would be to use the file based iscsi target. Would it shrink the file if data is removed from inside the file?

 

[danswartz]

Actually, I think I misspoke. The COW ZFS stuff should not apply (I don't think) to a zvol, since the internals of the zvol is NTFS (or whatever.) For zvol or file-based, if you fill up a certain amount of space, whether or not the client filesystem frees things, there will still be a certain high-water amount of zfs space being used.