• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

vMotion problems with same CPU different CpuIdInfo

J

jerkyjerk

n00b
Joined
Mar 30, 2011
Messages
8
I recently built out a pair of hosts to dig deeper into virtualization. The systems built out are identical just purchase a few weeks apart. I wanted to do a dry run on the first set of parts before committing to the build out of a second node. I used Xeon E3-1230 CPUs as the basis of both nodes. Surprisingly I found some slight differences between the two CPUs that is causing any vMotion operations to fail. It throughs an error that about "Host CPU is incompatible with virtual machine's requirements at CPUID level 0x1 register 'ecx'" so and so so forth. At first I thought maybe it was related to running nested hypervisors so I then tried vMotion'ing something one level up right on the host hypervisor level but the same error would get thrown. Since ESXi doesn't have /proc/cpuinfo available I used "vim-cmd hostsvc/hosthardware" to look for more details. After diff'ing the output from the two systems I found the two differences as seen below. Some other observations while I've been troubleshooting this. The other odd one is if I attempt to enable EVC (vSphere 4.1) and jump straight to the 32nm Core i7 "mode". it says one of the node's CPU doesn't support that mode. I thought that was odd given they are the same CPU. Any one see something like this before? I also would think it would be odd to have to use EVC for same make/model CPUs.

My next move is probably wiping the BIOS to defaults(again) and reinstall ESXi on both nodes.

Cheers,
Jeff



< hz = 3192747104,
---
> hz = 3192747584,
48,49c48,49
< hz = 3192747104,
< busHz = 99773347,
---
> hz = 3192747584,
> busHz = 99773362,
77c77
< ecx = "0001:0101:1011:1010:1110:0011:1111:1111",
---
> ecx = "0001:0111:1011:1010:1110:0011:1111:1111",
130c130
< memoryRangeLength = 8181706752,
---
> memoryRangeLength = 8181739520,
437c437
< ecx = "0001:0101:1011:1010:1110:0011:1111:1111",
---
> ecx = "0001:0111:1011:1010:1110:0011:1111:1111",
 
Virtualization is enabled in BIOS on all nodes involved?

Thats the only thing that comes to mind.

Same motherboard, cpu, BIOS versions and settings should = same ability.
 
Yes I have double checked the VT-x is enabled. I do have some 64-bit guest so that would have stood out at a possible culprit straight away. Everything is the same. I'll have to go over it closely when I get home. I'm pretty sure the bios and firmware versions are all in sync. I though the odd thing was the differences in the "feature" (for lack of a better term) ecx:

< ecx = "0001:0101:1011:1010:1110:0011:1111:1111",
---
> ecx = "0001:0111:1011:1010:1110:0011:1111:1111",

Both motherboards are Supermicro X9SCi-LN4F BTW.
 
Very odd indeed, I wonder if it is just a very minor difference but is still being reflected. What would be interesting to see, is if you did enable EVC at the level both those CPUs are at... if it started working. I'd like to know what the difference is, but you might never find out tbh.
 
I got a little curious ... I'm a n00bie at this stuff....
Seems like "AES" is what exist in one but not the other.

EVC = "Intel® Xeon® 32nm Core™ i7" exposes AES

Sources:
vMotion CPU Compatibility - Migrations Prevented Due to CPU Mismatch - How to Override Masks

Enhanced VMotion Compatibility (EVC) processor support

None of the above changes the fact that (assuming) perfectly matched CPU/Motherboard BIOS versions and setting being equal, yield differing results.

Honestly ... i'd be inside the case looking at all the CPU info and Motherboard info to verify.

It's crazy!
 
I setup a cluster with EVC enabled. I was able to move the one node into it with the level set to "Intel® Xeon® 32nm Core&#8482; i7" When I tried to move the second node into it threw an error:
<snip>
"The host's CPU hardware should support the cluster's current Enhanced vMotion Compatiblity mode but some of the necessary CPU features are missing......."
<snip>
I dropped the EVC level down one notch and was able to ad the node into the cluster and subsequently vMotion a guest.

As you noted, Networkz101, AES seems to be the feature exposed by that EVC level. I'll have to dig though the BIOS to see if something is turned off. I did re-set the BIOS to defaults but maybe it didn't reset things as expected.

At least it's a minor step forward. Back to digging when the kids go to bed.
 
It looks like the problem might have something to do with the motherboard. I'll have to unrack them and look closely for any hardware revision numbers they might have. They are running the same BIOS version but oddly one doesn't have the AES-NI option available in the bios. It looks like there is a newer BIOS revision available I'll give that a try and if that doesn't work I"ll take up the issue with Supermicro and see what support might say. For now I'll just disable the AES option on the one node to keep the two on the same level.
 
Below is more data to support the AES feature being the issue. After disabling AES-NI the only difference showing up is a slight speed variation. Thanks for letting me talk this out loud. The AES suggestion put me in the right direction sparing me from having to compare every BIOS options screen side by side looking for differences. I'll post any breakthrough i have in case someone else runs into a similar problem with these boards.

< hz = 3192747552,
---
> hz = 3192747616,
48,49c48,49
< hz = 3192747552,
< busHz = 99773361,
---
> hz = 3192747616,
> busHz = 99773363,
 
You must log in or register to reply here.
Back
Top