I've got a 3 TB datastore hosting a bunch of VMs, all of which have basically the exact same configuration (it's for student project boxes, so they all get the same base install). I'm currently showing on the array (ZFS) that I have a 26x de-dupe ratio and ~1.5 TB of usable space. At the same time, vSphere shows that there's ~275GB available on the datastore. I just don't understand why the array says it's only half full and vCenter thinks it's almost full. Do I need to set up a second target and start writing VMs into that, or is there some other way to get it to recognize the fact that there's still plenty of room?
![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
-
Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.
Too much deduplication?
- Thread starter Paladin21
- Start date
nicholasfarmer
Limp Gawd
- Joined
- Apr 19, 2007
- Messages
- 238
I think we need a bit more detail.
- Are all of the VMs thick or thin provisioned? If thick, did you eager zero?
- Are you storing the VMs vswap file on the storage or did you reserve the memory to drop the vswap waste?
- Any non vm storage on it? ISOs, backups, etc?
- Is your ZFS array presented as NFS storage or iSCSI?
Nick
- Are all of the VMs thick or thin provisioned? If thick, did you eager zero?
- Are you storing the VMs vswap file on the storage or did you reserve the memory to drop the vswap waste?
- Any non vm storage on it? ISOs, backups, etc?
- Is your ZFS array presented as NFS storage or iSCSI?
Nick
All VM's are thin provisioned, though they are all within 10GB or so of being full.
vSwaps are on storage, there isn't enough memory to do it in RAM, unfortunately.
Everything on this array is a VM.
iSCSI.
zfs list on the array shows 1.4TB availabe, vCenter shows 270GB. I wouldn't be concerned if the numbers are a little off, but that aren't even remotely close
vSwaps are on storage, there isn't enough memory to do it in RAM, unfortunately.
Everything on this array is a VM.
iSCSI.
zfs list on the array shows 1.4TB availabe, vCenter shows 270GB. I wouldn't be concerned if the numbers are a little off, but that aren't even remotely close
I figured out the issue I was having. I'm going to leave a brief description of the problem/fix in case anyone else has a similar issue.
The target LU was set up as a volume LU. vSphere was only reporting space remaining in the volume, instead of space remaining in the pool. I expanded the LU on the array to account for the deduplication space savings, ran the increase datastore wizard on vCenter, and now have extra space to mess with. I'm guessing that with the amount of space savings I'm running, I might have to do this process a couple of times to keep accounting for "extra" space. At some point I'll probably try setting it back up again as a raw LU, which I think should expand on its own? In any case, thanks for the help.
The target LU was set up as a volume LU. vSphere was only reporting space remaining in the volume, instead of space remaining in the pool. I expanded the LU on the array to account for the deduplication space savings, ran the increase datastore wizard on vCenter, and now have extra space to mess with. I'm guessing that with the amount of space savings I'm running, I might have to do this process a couple of times to keep accounting for "extra" space. At some point I'll probably try setting it back up again as a raw LU, which I think should expand on its own? In any case, thanks for the help.
You can do it a couple of times.
Then you'll start to run out of space in the pool, and everything will become a corrupted mess.
Dedupe works on NFS, where it's natively reported since we're talking file shares. iSCSI? You're just scattering landmines ahead of you
Then you'll start to run out of space in the pool, and everything will become a corrupted mess.
Dedupe works on NFS, where it's natively reported since we're talking file shares. iSCSI? You're just scattering landmines ahead of you
The array has 32GB of RAM, and 2x256GB Intel 520-series SSD's (can't think of the exact model off the top of my head) for both read and write cache. Drives are 300GB 10k SAS drives (Seagates, I think), and NIC is an Intel 10GBe adapter (x540). Performance is great, even with the de-dupe turned on. The array has to be high-performance as it hosts a bunch of VMs for database programming classes, so there's ~30VM's turning off and ~30 more turning on in a 10 minute window between classes. Proof of concept version 3(4?) years ago ran a Drobo Pro (since it was the cheapest certified solution), and the VM's were so slow they *might* boot to a desktop before the class was over, but usually wouldn't.
I'm going to assume you have 128K set as your zfs recordsize.
3TB volume = 3221225472 KBytes
3221225472 / 128K (your block size) = 25165824 total blocks in the volume
25165824 total blocks * 320bytes (individual dedupe pointer size per block) = 7.5GB after conversion back to GB.
meaning, if your volume is defaulting to 128K record size 7.5GB of that 32GBs of ram is being eaten by the dedupe table.
if say you're using 64K record size, that is 17GB of ram. 32K would require 34GB of ram, etc etc. I don't know what you're record size is but it is entirely possible the dedupe table has fallen out of ram (or partially fallen out). if that happens then you're having to read the DDT off spin disk which is going to be painfully slow.