Thin Provisioning + Virtualization

[Dark Shade]

Currently we are using thick provisioned LUNs in our iSCSI SAN network and we would like to investigate thin provisioning our infrastructure. It's 99% homogeneous installs of CentOS 6, so the operating systems on these virtual guests across the board are nearly identical, the only difference between them are some applications or databases.

The overall documentation for the actual use cases is sparse, most Google searches has returns generic information about thin provisioning and not specific answers to questions, such as:

1. If I transfer a thick-provisioned LUN to a thin-provisioned LUN, do I gain the benefits of thin provisioning, or do I need to install from scratch? (dd if=/dev/olddisk of=/dev/newdisk)
2. Do I simply configure thin-provisioned LUNs to have a 'maximum size' on the SAN? I assume the FS would require this.
3. We have significant monitoring tools, so we watch current disk usage very well. What is the general rule-of-thumb for ensuring disk usage doesn't go past a certain % of a thin-provisioned RAID Group? 60%? 80%?
4. Is there a large performance difference due to the thin-provisioned pool being allocated on the same platters, or does the SAN smart-allocate this around the disks so the same ones aren't being hammered all the time while others idle?

Those are the main questions that pop in my head for now, I may have more soon.

Please correct me if I am wrong about how I have understood thin-provisioning:

I create multiple thin-provisioned LUNs. I install the same exact OS on each one via any method (kickstart, manual, etc.). For this example I am assuming the OS is going to take up 3GB. The data that is used up for these installs is 'pooled' amongst the LUNs totals 3GB, and any differences between these LUNs does not get pooled, such as if one machine turns into an apache webserver, and another is a mysql database. Each of these machines are virtualized guests (KVM) sitting on a host that connects to the SAN via iSCSI.

School me if I am wrong! Thanks!

 

[patrickdk]

You are defently confused.

Somehow you confused dedup with thinprovisioning.

The only thing thin does is not say empty diskspace is used. but that EMPTY space can be used by another vm.

It does nothing about if something is the same.

 

[Dark Shade]

So suffice it to say, thin vs. thick, thin will let you reallocate unused space that was previously used, and thick takes the space and never lets it go?

Man maybe I should just suggest we use ZFS

 

[patrickdk]

No.

Thin lets you reallocate unused space that was NEVER previously used.

Thick reserves the space, used or not.

Dedup makes thick and thin basically the same, or even, just using compression.

 

[rsq]

I use a ZFS iSCSI server with thin provisioning and cloning. No dedup. I am extermely happy with that arrangement.

More details:
1. First step was to create a thin provisioned ZVOL (zfs create -s -V 200G /tank/base) and share that via iSCSI.
2. On the VM server, I created a VM on that LUN and installed my base OS (Ubuntu server 12.04). I just used the default install, the common base for all my VMs. I installed all the updates.
3. After the base install, I created a snapshot for the /tank/base zfs.
4. I created 10 clones of the snapshot, and shared all of them via iSCSI. I created a VM for each new LUN.
5. I customized VM per VM to make them do what I want. Usually, this entails having some config files deployed and setup of a website on apache.

At this point, I have 11 VM's, each thinking they have a disk of 200G, and my actual used diskspace is 6Gig.

I will now observe the disk usage, and if it gets to 80%, I add more drives to the pool (I use striped mirrors, so only 2 disks per expansion cycle)

This setup allows me to spin up a new VM in 3 seconds (I have a script)

The really cool thing is that I can keep the base install up to date, and after a bunch of updates, I can make a new snapshot, which is then used for clones. The clones start life as up to date as the base install was at the moment of cloning.

There is another extremely nice bonus to all this: The whole base VM is 5 gigs large, and its blocks are used by all clones. This means that all of the base VM is served from the ARC (from RAM) and the disks are more or less free to deal with the VM operational requests.

BTW: I decided against using dedup because of the RAM requirements. I prefer to use my RAM for ARC.

 

[danswartz]

Different strokes, I guess. I don't mind thin for NFS, but I don't like thin for iSCSI (vmfs), since vsphere will think it has a certain amount of space to use, and if the underlying SAN storage fills up, vsphere will unexpectedly get fatal write errors. I know a guest can get a write error with an NFS backed disk, but that just hoses that one guest. I don't know if anything horribly bad can occur if a VMFS metadata operation fails this way, but I'd rather not find out...

 

[rsq]

Let me clarify:
I rolled my own VM server based on KVM. My setup is untested with VMWare.

I also use one LUN per VM. I looked into Xen and Proxmox, and they both want to run LVM on the whole LUN, and do their own snapshotting and such. I definitely did NOT want LVM because everything it can do can be done better by ZFS.

I can't say anything meaningful about VMware because the accountant at work looked at the licensing terms and prices and said: "Oh hell no!", so I did not investigate further.

 

[danswartz]

Oh, sorry, so many people commenting, I misunderstood your use case.

 

[Thuleman]

I can't say anything meaningful about VMware because the accountant at work looked at the licensing terms and prices and said: "Oh hell no!", so I did not investigate further.

That's why you don't leave it up to accountants. VMware is by orders of magnitude superior to all other virtualization solutions because you save on payroll when you deploy VMware.

Everything you could ever dream of doing with your deployment has already been developed, tested, and released with VMware. The amount of time you save vs monkeying around of KVM, having no one to call when something goes wrong, etc. etc., more than makes up for the VMware licensing costs. Also for the SMB market there is vSphere Essentials (6 sockets + vCenter) for $650 MSRP, and vSphere Essentials Plus (recommended, 6 sockets + vCenter, plus a bunch of useful stuff for production) for $5.4k MSRP. Depending on your VAR, or whether you are in Government/Academia, you can get some decent discounts on all of this. 5k for licenses in the first year is the equivalent of 50 hours of payroll (including benefits). I can almost guarantee that many outfits spend hundreds of combined hours screwing around with KVM and still not getting the ease of operation VMware provides.

Also, thin provisioning is the devil.
Also, [H] Virtualized Computing is where you will likely get virtualization related posts in front of more expert eyeballs.

 

[danswartz]

This.

 

[Nex7]

I'm glad rsq already brought it up.

If what you have is a bunch of nearly or entirely homogeneous VM's, especially if they're not permanent, or are semi-permanent in that you can replace them when doing major upgrades to the OS, etc, and you either do not need HA storage or your total number of VM's is less than, say, 100, then ZFS cloning is the most incredible, magical, cost-saving fairy dust you've ever seen in your life.

On the other hand, that's a few if's, and if they're heterogeneous or need to be permanent for many years including across major OS updates, or you have many 100's of them and require HA storage, it's probably nearly completely useless to you.

I like Thuleman's "use what works" attitude - though I question being quite that harsh on KVM paired with a decent management layer (some of which are free) if your use-case isn't overly demanding or complicated. It gets you most of the way there at $0 out the door, and in some businesses, even extra manpower can be acquired at a greater cost than the licensing (weird, but I've seen it -- hire someone? yes -- pay VMware licensing fees? no; and that's a decision from management). And you definitely can't sit there and say Xen is lightyears behind VMware (behind? maybe, but lightyears? please, no, they've been around the block a few times themselves), and their equivalent to vSphere Essentials is like half the price or less, so depending on how price-conscious you are, it might appeal. If the only feature you're after is 'vMotion', then XenServer includes that one for free, even.

... that all said, if I was needing to pick specifically between XenServer Essentials and vSphere Essentials Plus, I'd probably tell the boss to spend the double for VMware. I don't do much of it, because all my own stuff is KVM, but when I do find myself managing paid hypervisor infrastructures, I feel better about the vSphere ones than the Xen ones, and I can't really tell you a specific why, I just do.

 

[brutalizer]

I use a ZFS iSCSI server with thin provisioning and cloning. No dedup. I am extermely happy with that arrangement.

The really cool thing is that I can keep the base install up to date, and after a bunch of updates, I can make a new snapshot, which is then used for clones. The clones start life as up to date as the base install was at the moment of cloning..

How do you mean? If you upgrade the base vm, then the the clones are not upgraded. They need to be upgraded as well. Could you clarify?

 

[kac77]

The amount of time you save vs monkeying around of KVM,

Whoa. I give VMWare all the credit in the world for it's management tools, but setting up KVM is breeze. I can go from no VM solution to a VM running and onto my next project within 5 minutes (and that's because I have to reboot the machine).

 

[Thuleman]

Whoa. I give VMWare all the credit in the world for it's management tools, but setting up KVM is breeze. I can go from no VM solution to a VM running and onto my next project within 5 minutes (and that's because I have to reboot the machine).

Sure, except that "setting up KVM" doesn't mean you have set up all the functionality VMware provides out of the box. Getting a VM running isn't the time consuming part.

 

[rsq]

How do you mean? If you upgrade the base vm, then the the clones are not upgraded. They need to be upgraded as well. Could you clarify?

Brutalizer, The clones are not updated automatically if I update the base VM. I need to update the clones individually, which makes them diverge from the base with age.

I'm sorry if I gave the impression that updating the base updates the clones as well.

That's why you don't leave it up to accountants. VMware is by orders of magnitude superior to all other virtualization solutions because you save on payroll when you deploy VMware.

Well, I work in a small company, there is just me to maintain the systems. So using KVM or VMware doesn't factor in the payroll. I maintain 4 servers, and +/- 30 VM's in production + a lab setup. I am a programmer by trade (the sysadmin thing is only part of my job), so writing some scripts and using free stuff is in my case much superior than paying for an of the shelf solution. I don't agree that VMware is superior, not for my application.

I would also state that there are much more small companies like mine then fortune 500's that run VMware on every server + their toasters... Maybe that's why solutions like Proxmox are so popular.

 

[kac77]

Sure, except that "setting up KVM" doesn't mean you have set up all the functionality VMware provides out of the box. Getting a VM running isn't the time consuming part.

Most people don't need "all of the functionality" which is the point.