VMware sockets vs cores and Co-Stop

K

KapsZ28

2[H]4U
Joined
May 29, 2009
Messages
2,114
I'm sure this is something that has been asked tons of times. Is it better to allocate more virtual sockets as opposed to more cores per socket. That is part of my question, but also wondering if it contributes to a high Co-stop percentage.

In this particular case we have 22 Fedora VMs with 2 cores each and they are running MySQL. The person that set it up selected 2 sockets. Normally I would go with 1 socket and two cores per socket.

These VMs are constantly alerting on percentage of Co-stop in which case the recommendation would be to remove one of the CPU cores. What I am trying to determine is if that is really what is needed to fix it, or if having two sockets compared to one socket with two cores is creating additional contention and causing these alerts.

There appears to be a similar trend between Co-stop and CPU Ready. Although no other VMs are complaining about CPU Ready and there appears to be enough resources.

What are your thoughts?
 
Honestly, I've not had much trouble with either, I tend to go more CPU lite/core Heavy with mine, but never seem to have any CPU issues like this. If you can shut down one or 2, I would, and switch them from 2cpu/1core to 1cpu/2core, and boot them back up, to the OS, nothing should change, since the number of processors it detects should be the same. Then you can see if the problem persists.
 
This article may be of use.

Based on it, I believe that more sockets vs more cores per socket will provide better performance, but it may be negligible in some or most cases or depend on your specific configurations. Probably a bigger concern when you are configuring VMs with more than a few sockets/cores.
 
Also, how about odd vs even number of cores? Are there are issues with having 3, 5, or 7 cores rather than an even number?
 
Nope, Odd number CPU's are great, better than rounding up and making the scheduler work harder.
 
If you're talking about the actual functionality of giving more cpu's via cores or cpu's to your VM's from the VM edit tab, I believe it's been stated by VMware that it does not matter, it's all cores at that point. The VMware scheduler will take what it can get when it can get it.
The use of 1cpu x 2cores allows you to kind of cheat with operating systems that may either have licensing limitations, ie. Windows 2003/8 Standard addition. (tricking the os to have more than 4 cpu's).
The reality is that you get the same thing either way.
 
I prefer 1-2 cores per socket because when you hot add resources you cant add cores to a socket only add another so if you need a touch more CPU power but (1 core) but have 1 4 core CPU you have to add 4 more cores or shut the VM down to only add one.
 
The only time I have found that it matters is when you are dealing with software licensing that charges you per socket. Some applications also are more optimized to work with multiple sockets rather than multiple cores (although these are usually older legacy apps, not new ones). Other than that, 1 single socket = 1 single core share of MHz. in my experience.
 
It matters when you take NUMA into account. If you're dealing with a single socket host, it truly doesn't matter but as you start to deal with two, four, etc. socket hosts then it's better to provision vCPUs with multiple vCPU sockets rather than cores so the CPU scheduler can better distribute the vCPUs across NUMA nodes. It makes a noticeable performance impact if you don't.
 
Child of Wonder said:
It matters when you take NUMA into account. If you're dealing with a single socket host, it truly doesn't matter but as you start to deal with two, four, etc. socket hosts then it's better to provision vCPUs with multiple vCPU sockets rather than cores so the CPU scheduler can better distribute the vCPUs across NUMA nodes. It makes a noticeable performance impact if you don't.
Click to expand...

Ok, well then another example. Server is two sockets. VM has 8 vCPU. I assigned two sockets with four cores each. Would changing it to 8 sockets with a single core make a noticeable improvement?
 
KapsZ28 said:
Ok, well then another example. Server is two sockets. VM has 8 vCPU. I assigned two sockets with four cores each. Would changing it to 8 sockets with a single core make a noticeable improvement?
Click to expand...

It depends on how the NUMA nodes on your server are setup. VMware has a good article on this let me see if I can find it.
 
ChildofWonder said it best.
To add to that, if you select only sockets, IE: 8 sockets and 1 core or 4 socket and 1 core, the scheduler takes this as a "default" config and is able to manage the scheduler the best possible way in terms of NUMA. All socket and one core gives the scheduler flexibility. If you "demand" a socket and core selection then that is locked in when the scheduler runs.

In terms of 1 socket 2 core or 2 socket 1 core, in terms of numa, it's the same and you should not see any change. On your eight vCpu VMs that must cross numa boundaries, the scheduler cannot change anything to make it place better.

You may find better performance by dropping those eight core VMs to four core and have them live in the same numa boundary.

Nicholas Farmer
 
nicholasfarmer said:
ChildofWonder said it best.
To add to that, if you select only sockets, IE: 8 sockets and 1 core or 4 socket and 1 core, the scheduler takes this as a "default" config and is able to manage the scheduler the best possible way in terms of NUMA. All socket and one core gives the scheduler flexibility. If you "demand" a socket and core selection then that is locked in when the scheduler runs.

In terms of 1 socket 2 core or 2 socket 1 core, in terms of numa, it's the same and you should not see any change. On your eight vCpu VMs that must cross numa boundaries, the scheduler cannot change anything to make it place better.

You may find better performance by dropping those eight core VMs to four core and have them live in the same numa boundary.

Nicholas Farmer
Click to expand...

And this NUMA stuff works fine with odd number of cores?
 
KapsZ28 said:
And this NUMA stuff works fine with odd number of cores?
Click to expand...

Yes, but with larger VMs you would want to be using 6 or 12 core physical processors in your hosts if you are going to be using odd numbers of vCPU.
 
SGalbincea said:
Yes, but with larger VMs you would want to be using 6 or 12 core physical processors in your hosts if you are going to be using odd numbers of vCPU.
Click to expand...

These servers are running dual Intel E5-2680 v2 processors and the VMs I am thinking about changing would be from 8 to 7 vCPU.
 
The "best practice" is to only increase the socket count on a VM and leave the cores at 1. This lets the scheduler handle the placement. The only reason to change the core digit is if you are running an operating system that cannot handle the socket count. Example: try to set the socket count to eight on a windows 2003 standard edition OS..

Second thing is to attempt to stay within a numa boundary. Meaning, if you have quad core CPUs, then try to keep the VMs under four vCpus. (Four socket, one core)
In rare instances we do need to make VMs more vCpus the. Physical count per processor but with some testing, you may find staying within the numa boundary, the VM is able to use less cpu when attempting to processes.

I've had many examples where an application owner said their app was maxing out at 100% CPU and they needs more cores. Magically their "report" started to take twice as long to run when the vcpu count was increased because it caused numa or even scheduling issues.

Nicholas Farmer
 
You know, I wish that had mentioned this in my VMware class. Thanks for the info guys. I guess I have a lot of changes to make.
 
Some more info:

Also keep in mind VM memory.
If you are running a two socket system (two quad core cpus for example) and if both physical CPUs have 16GB of ram each.. (32GB system total) assigning a VM more than 16GB of ram will cause numa issues. (more than 14GB really because of overhead)

KapsZ28 - You said you have E5-2680 v2 CPUs.. These ten core CPUs should be able to handle an eight vCPU vm without issue. Setting the eight vCPU VM as 8socket 1 core, 4X2, 2X4, or 1X8 shouldn't matter unless you are over committing the host. I would check CPU ready and wait times and then tweak the socket/core settings and check again for any improvement. If nothing is noticed on one VM, I wouldn't expect a difference on others.

Last note on Numa and vCpu settings. There are some flavors of AMD processors that were listed as a quad core CPU but was actually two - Two core CPUs smashed together. Bulldozer I think? I forget...
These Processors had numa issues within the socket because it had to use a slower bus to transfer memory states between the processors within the same socket (if your VM was a quad vcpu for example.)

Lucky, that is old tech and shouldn't kill too many VM Admins today...

Nicholas Farmer
 
nicholasfarmer said:
Some more info:

Also keep in mind VM memory.
If you are running a two socket system (two quad core cpus for example) and if both physical CPUs have 16GB of ram each.. (32GB system total) assigning a VM more than 16GB of ram will cause numa issues. (more than 14GB really because of overhead)

KapsZ28 - You said you have E5-2680 v2 CPUs.. These ten core CPUs should be able to handle an eight vCPU vm without issue. Setting the eight vCPU VM as 8socket 1 core, 4X2, 2X4, or 1X8 shouldn't matter unless you are over committing the host. I would check CPU ready and wait times and then tweak the socket/core settings and check again for any improvement. If nothing is noticed on one VM, I wouldn't expect a difference on others.

Last note on Numa and vCpu settings. There are some flavors of AMD processors that were listed as a quad core CPU but was actually two - Two core CPUs smashed together. Bulldozer I think? I forget...
These Processors had numa issues within the socket because it had to use a slower bus to transfer memory states between the processors within the same socket (if your VM was a quad vcpu for example.)

Lucky, that is old tech and shouldn't kill too many VM Admins today...

Nicholas Farmer
Click to expand...

Current CPU Ready is less than 1%, but I also want to plan for the future since we will be continuing to add more VMs. You mentioned memory. These VMs with 8 vCPU also have 32 GB of memory allocated. Is that going to be an issue?
 
These VMs with 8 vCPU also have 32 GB of memory allocated. Is that going to be an issue?
Click to expand...

Only if the system has less than 32GB of ram attached to a CPU Socket.
IE - A single proc on the motherboard you will need at least 32GB+overhead on RAM in the system.
Two CPUs on the motherboard, you need 32GB+ overhead of RAM in each CPU bank (over 64GB on the motherboard.)

Nicholas Farmer
 
You know, I am not really sure about this socket vs cores and NUMA stuff. I changed 1 out of 8 VMs to use 8 sockets instead of 2 sockets with 4 cores each and that VM shows the highest ready time. Granted, the ready time is insignificant at the moment, but it is still the highest.

8 sockets - average of 450ms ready time.
2 sockets, 4 cores each - average of 136ms ready time.

Those are both on the same ESXi host. Other identical VMs have less than 2ms of ready time.
 
You must log in or register to reply here.
Back
Top