• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

New Home Lab - VMWare or Hyper-V

P

PhoenixC5

Limp Gawd
Joined
Jun 7, 2004
Messages
335
So I'm finally building my new white box home VM lab. First off, I'd love to get opinions on my hardware. Does anyone see any issues with the following configuration:
CPU:
Xeon 2620 V2 x 2 [http://www.newegg.com/Product/Product.aspx?Item=N82E16819116934]

CPU Cooler:
Noctua NH-U9DX I4 [http://www.newegg.com/Product/Product.aspx?Item=13C-0005-00001]

Motherboard:
ASRock EP2C602-4L/D16 [http://www.newegg.com/Product/Product.aspx?Item=N82E16813157350]

Memory:
16GB Kingston DDR3 1600 Registered ECC x 8 (128GB Total) [http://www.newegg.com/Product/Product.aspx?Item=N82E16820239892]

Case:
Norco 4224 [http://www.newegg.com/Product/Product.aspx?Item=N82E16811219038]

Power Supply:
Thermaltake Toughpower 1200W [http://www.newegg.com/Product/Product.aspx?Item=N82E16817153054]

RAID Controller:
Areca 1880i [http://www.newegg.com/Product/Product.aspx?Item=N82E16816151071]

SAS Expander:
HP SAS Expander [http://h18004.www1.hp.com/products/servers/proliantstorage/arraycontrollers/sas-expander/index.html]

SSD Array:
Samsung 840 Pro 256gb SSD x 4 [http://www.newegg.com/Product/Product.aspx?Item=N82E16820147193]

Primary HDD Array:
Hitachi 7K3000 2TB 7200RPM Hard Drives x 8 [http://www.newegg.com/Product/Product.aspx?Item=9SIA0AJ1XS5290]

Secondary HDD Array:
Seagate Barracuda LP 2TB 5900RPM Hard Drives x 6 [http://www.newegg.com/Product/Product.aspx?Item=N82E16822148413]

Another SSD:
Samsung 840 500GB SSD [http://www.newegg.com/Product/Product.aspx?Item=N82E16820147190]

So assuming all of that looks good, next up is the big decision. Do I go VMWare or Hyper-V. I've been planning the entire time to go VMWare, until I started researching the beginning. My biggest concerns with VMWare are that the free version has vCPU limitations, number of VM limitations, and management constraints (vCenter definitely isn't free). So short of spending quite a bit of money on software after spending plenty on hardware, which will leave me in trouble with my wife...I started looking at alternatives. From what I can tell, Hyper-V should perform well, is free, and has none of those limitations. But I still have trouble wanting to use it. I've used older releases of Hyper-V in the past due to hardware compatibility issues, but this server is basically all actual server hardware, no real desktop hardware (a few drives).

So what will I be using this server for? A few things. My lab will consist of the following basics:
Windows AD Box
Microsoft SQL Server Box
Some type of Mail Appliance to use as an SMTP relay...or I may just add that to my AD box
OpenFiler for storage needs (NFS and FTP primarily)
A bunch of other servers for an Oracle suite of products name "Hyperion"

So the vast majority of my clients use ESXi, so I feel drawn to that as a solution. But at the same time, the 8 vCPU limit could have an impact on the performance of one or more of my VM's when I do some load balance testing and benchmarking.

Okay...that's enough 2am rambling for now. Opinions? Guidance? Questions? Comments? Concerns?

Thanks in advance!

I lied...one more thing. The biggest performance implications for this lab will be around a database product named Essbase. That will be the only VM that I may need more than 8 vCPU's. Additionally, the SSD RAID array will be used as the primary storage for the Essbase box, and nothing else. I will likely use the 500GB Samsung 840 (non-pro) to store the main hard drives of the VM's. May eventually add another one and do RAID 0 with that as well to give me more space and performance. The HDD arrays will likely be for backup, file storage, FTP, and other related things.
 
Last edited:
If you expect to assign a lot of vCPUs (more than 24 vCPUs total), Hyper-V has an advantage there as it handles multiple vCPUs better by simply treating them the same as Windows always has with process threads while ESXi uses an improved Unix method of CPU co-scheduling. CPU Ready time, a bane for VMware admins, isn't really an issue in Hyper-V. However, I doubt you'll be running enough vCPUs hard enough for that to be an issue.

Since you're building a 2 socket server be mindful of your NUMA topology. Even though both hypervisors will tell you they have 24 logical cores (12 physical cores x2 with HT) DO NOT assign more than 12 vCPUs to any VM or more than 6 vCPUs if you disable NUMA spanning on a VM. HT helps with distributing many vCPUs from many VMs across the physical cores but it seriously hampers performance when a single VM thinks it has, for example, 24 vCPUs or 24 physical cores yet the host has only 12 physical cores/24 logical cores. Logical core does not equal physical core! I have seen people do this and scratch their heads when the VM performs like shit. For this same reason, I'd advise against giving a single VM more than 6 vCPUs (the number of physical cores you have in a single NUMA node) unless absolutely necessary.

ESXi will be a better solution for your Oracle and Linux deployments since Linux has been running virtual in VMware since day 1 while Linux support has been sketchy in Hyper-V. BSD releases cannot be virtualized in Hyper-V at all. ESXi will also be easier to upgrade while performing an in place upgrade of Windows is sketchy at best and not recommended.

As you mentioned, if you use the free Hyper-V you have no limitations on VM resources while you do in free ESXi. If you ever want to add a second host, you'll need AD to cluster your Hyper-V hosts while ESXi does not need AD but requires vCenter (yet, you should still use AD).

In the end, I don't see anything that's a deal breaker for either hypervisor. I think it just comes down to which one you want to learn more about.
 
Interesting. So in my case, I have one VM that I'd like to give as much CPU and memory to as possible. 6 vCPU limit would be unfortunate. I have 12 physical cores, but I can't cross over without running into performance issues?
 
PhoenixC5 said:
Interesting. So in my case, I have one VM that I'd like to give as much CPU and memory to as possible. 6 vCPU limit would be unfortunate. I have 12 physical cores, but I can't cross over without running into performance issues?
Click to expand...

Depends. If in ESXi you give your VM 8 virtual cores and 1 virtual socket for 8 vCPU total, you'll hit performance issues. It's best to give the VM 8 virtual sockets so ESXi can properly distribute the vCPUs across NUMA nodes.

ESXi will also expose the underlying physical NUMA architecture to the VM OS so if the guest OS and applications are NUMA aware, they'll benefit from that.

Here's more info:

http://blogs.vmware.com/vsphere/2012/02/vspherenuma-loadbalancing.html

Here's performance data on virtual sockets vs virtual cores in relation to NUMA node sizing:

http://blogs.vmware.com/vsphere/201...t-performance.html?utm_source=longwhiteclouds

I'm not saying you have a 6 vCPU "limit" just that if you can avoid sizing a VM with more than 6 vCPUs, try to do so, as well as not giving a VM more than 64GB of RAM (each NUMA node on your host will be 6 physical cores and 64GB RAM). That way NUMA doesn't even come into play.
 
If you can get your hands on a Server 2012 (R2) Datacenter then you have unlimited number of Windows Server VMs to play with without the need to license any of them. By far my favorite part of Hyper-V.

I've been using Hyper-V for about a year at home and my biggest problems are:

#1. Lack of appliance support. Meaning, people seem to be writing cool appliances that are packaged for VMWare or VirtualBox or Xen but not Hyper-V.

#2. Lack of Linux support. Most of my VMs are Windows Servers but I have a few Linux boxes and sometimes I have to fight with stuff like the network adapters to get them to work right.

#3. Frustrating management. Some stuff is just confusing at first, like clustering or Kerberos delegation. Some stuff requires a VMM Server and even just setting up VMM can be a pain. Sometimes I run into strange permissions issues too.

Now if you're a Windows guru then some of this stuff will be totally unimportant to you, as when I started playing with it I knew nothing about Hyper-V.

As far as NUMA goes Hyper-V in Server 2012 R2 is NUMA aware, lets you set custom values etc and all that good stuff.
 
Child of Wonder said:
If you expect to assign a lot of vCPUs (more than 24 vCPUs total), Hyper-V has an advantage there as it handles multiple vCPUs better by simply treating them the same as Windows always has with process threads while ESXi uses an improved Unix method of CPU co-scheduling. CPU Ready time, a bane for VMware admins, isn't really an issue in Hyper-V. However, I doubt you'll be running enough vCPUs hard enough for that to be an issue.

Since you're building a 2 socket server be mindful of your NUMA topology. Even though both hypervisors will tell you they have 24 logical cores (12 physical cores x2 with HT) DO NOT assign more than 12 vCPUs to any VM or more than 6 vCPUs if you disable NUMA spanning on a VM. HT helps with distributing many vCPUs from many VMs across the physical cores but it seriously hampers performance when a single VM thinks it has, for example, 24 vCPUs or 24 physical cores yet the host has only 12 physical cores/24 logical cores. Logical core does not equal physical core! I have seen people do this and scratch their heads when the VM performs like shit. For this same reason, I'd advise against giving a single VM more than 6 vCPUs (the number of physical cores you have in a single NUMA node) unless absolutely necessary.

ESXi will be a better solution for your Oracle and Linux deployments since Linux has been running virtual in VMware since day 1 while Linux support has been sketchy in Hyper-V. BSD releases cannot be virtualized in Hyper-V at all. ESXi will also be easier to upgrade while performing an in place upgrade of Windows is sketchy at best and not recommended.

As you mentioned, if you use the free Hyper-V you have no limitations on VM resources while you do in free ESXi. If you ever want to add a second host, you'll need AD to cluster your Hyper-V hosts while ESXi does not need AD but requires vCenter (yet, you should still use AD).

In the end, I don't see anything that's a deal breaker for either hypervisor. I think it just comes down to which one you want to learn more about.
Click to expand...

No more resource limits on free, IIRC.
 
lopoetve said:
No more resource limits on free, IIRC.
Click to expand...

There is still a limit of 8 vCPU per guest, but that really isn't a big deal since few need more than that for a home lab or test box.

Plus the Virtual Hardware 10 features like 2TB+ VMDKs, enhanced vGPU features, etc. are only available using the vCenter web client.

Again, not big deals.
 
MysticRyuujin said:
#2. Lack of Linux support. Most of my VMs are Windows Servers but I have a few Linux boxes and sometimes I have to fight with stuff like the network adapters to get them to work right.
Click to expand...

Not sure if that's accurate anymore. The fundamental problem with this was to not use the legacy network adapter but the Hyper-V aware adapter, but that solution is at least a three years old and has since worked fine afaik.

Here's what MS says to do to avoid Linux VM issues:
- Use static MAC addresses with failover clustering
- Use Hyper-V-specific network adapters, not the legacy network adapter.
- Use I/O scheduler NOOP for better disk I/O performance.
- Add “numa=off” if the Linux virtual machine has more than 7 virtual processors or more than 30 GB RAM.

See: Best Practices for running Linux on Hyper-V
 
Child of Wonder said:
There is still a limit of 8 vCPU per guest, but that really isn't a big deal since few need more than that for a home lab or test box.

Plus the Virtual Hardware 10 features like 2TB+ VMDKs, enhanced vGPU features, etc. are only available using the vCenter web client.

Again, not big deals.
Click to expand...

all being fixed soon too ;) Or, at least, most...
 
Just a couple of more thoughts, one it's nice to be able to do something with the box besides host vms. Of course, if you are trying to learn Vmware or Hyper-V, pick the one you want to learn or suits the majority of your vm types.

Another option is running KVM in your favorite flavor of Linux. It's at least worth checking out.
 
If you know windows well, I really like Hyper-V. 2012 is a ton better than 2008.

I started with ESXi, moved to XenServer, then moved to Hyper-V. So far I have been using Hyper-V for the longest in my lab and really enjoy it. The integration with a windows environment is very slick.

I have 2 hosts, unclustered, mostly Windows VMs with a few linux VMs. The replication feature is fantastic for backups (automatically snapshots running servers to the other host for a cold spare).
 
Child of Wonder said:
As you mentioned, if you use the free Hyper-V you have no limitations on VM resources while you do in free ESXi.
Click to expand...

Maybe I am misremembering here, but didn't they remove all resource limits in ESXi 5.5?

edit:

Ahh, just saw the 8 vCPU lmit per host. I've never run into that one myself :p
 
Zarathustra[H];1041063766 said:
Maybe I am misremembering here, but didn't they remove all resource limits in ESXi 5.5?

edit:

Ahh, just saw the 8 vCPU lmit per host. I've never run into that one myself :p
Click to expand...

per-vm, not per-host. :p
 
If you are mostly a Microsoft shop then running hyper-v is a no brainer. All of the currently supported Microsoft operating systems are enlightend out of the box and should perform well even without updating the guest tools.

As for the Linux side of things Microsoft does have hyper-v drivers in the kernel and most major distro's should be supported out of the box.
 
I run Hyper-V on a couple servers I migrated from ESXi, I have one Windows Server 2008 VM for my DC, rest of the VMs are Linux, mostly CentOS 6.x bunch of pFSense boxes and one or two Ubuntu boxes, works great, happier with Hyper-V than VMWare ESXi personally

plus it's free, can do live migrations and all sorts of fun stuff, transferring files to/from the host is fast (unlike ESXi which is painfully slow) plus you can install the free 5nine manager on the host and access your VMs from the host itself which is kind of nice sometimes
 
pfsense runs ok on hyper-v? That's been the one that has kept me from looking more that direction.
 
dave99 said:
pfsense runs ok on hyper-v? That's been the one that has kept me from looking more that direction.
Click to expand...

There was a release on the forum that was supposed to be stable on HyperV, but 2.2 is coming out in few months. That is based on the newer version of FreeBSD which has much better hyperv support and should work with no issues. I'm anxiously waiting for that release as well since we use all windows machines at work.
 
I like Hyper-V for being able to clone VM's cleanly. Just copy/paste the VHD file, sysprep, boom, new machine. For testing purposes its nice, the main reason why I switched to Hyper-V from ESXi.
 
ESXi
Copy or move a VM folder, import to inventory (a mouse right click on the .vmx file in the ESXi filebrowser) and start the VM.
Answer whether you have moved or cloned the VM, thats it.
 
_Gea said:
ESXi
Copy or move a VM folder, import to inventory (a mouse right click on the .vmx file in the ESXi filebrowser) and start the VM.
Answer whether you have moved or cloned the VM, thats it.
Click to expand...
I'll have to give that a shot.
 
Excellent thread! Some great info here. I use primarily Hyper-V in my home lab, though I do also have an ESXi box. I've used ESX for years at work, but actually prefer Hyper-V for home as well. Most of my machines are Windows guests, so a bit of a no brainer there. Heck, it's just nice to have so many options these days. I remember when installing a new ESX host would take most of a day and a book worth of custom configuration! Good times.
 
_Gea said:
ESXi
Copy or move a VM folder, import to inventory (a mouse right click on the .vmx file in the ESXi filebrowser) and start the VM.
Answer whether you have moved or cloned the VM, thats it.
Click to expand...

Just to expand on that - with my Server 2008 VMs I'll take a new install, download all the security updates, set basic things like NTP and such, then run sysprep (generalize, shutdown). Once this VM shuts down, you just leave it as a baseline or template. Then follow Gea's instructions and you have a new clone ready to go!

After booting up the new VM you set your new admin password. From there I'll set a static IP and join it to a domain if I have one.
 
_Gea said:
ESXi
Copy or move a VM folder, import to inventory (a mouse right click on the .vmx file in the ESXi filebrowser) and start the VM.
Answer whether you have moved or cloned the VM, thats it.
Click to expand...

but VMFS is sooooooooooo slow, like, unbearable slow, I transfer files to/from hyper-v at full speed 800~900mbits easily, with VMWare I was lucky if I could get 100~200mbits on the exact same hardware
 
You must log in or register to reply here.
Back
Top