ESXi 5.5... Ugh!

[TeeJayHoward]

I "upgraded" my ESXi rig from 5.1 to 5.5 recently. The first thing I noticed was how incredibly sluggish everything ran. Apparently, ESXi can no longer handle a ZFS pool shared via NFS unless you disable sync. That only took 3 hours to figure out. Next, I noticed that my backup server wasn't working. Turns out that passing through the SATA controller on my X9SCM no longer works at all. Oh, sure, it passes through to the OS, but it can't be used. Windows gives it a code 10, and Solaris can't see anything plugged in to it. I tried it in the fat client and in the web client.

Which, by the way, who's genius idea was THAT hunk of junk?

A) It creates VMs only usable by the web client.
B) It takes over control of all VMs on that host so that you can't modify anything EXCEPT through the web client.
C) It does B even if the vCenter Server Appliance is turned off. The only way to fix it is to reboot the host.
D) It utilizes 4GB+ RAM on the resource-limited server instead of 65MB of RAM on a client.
E) It takes FOREVER to do anything. Every time I click a button, I'm forced to wait several seconds for the page to update. (And this when it's the only VM running!)
F) You can't open the console unless the VM is turned on - So much for hitting keys early in the boot process!

I am VERY thankful I decided to install to a different USB stick instead of overwriting my 5.1 install. Fixing my backup server was as simple as plugging in the old stick, creating a new VM, passing through the controller, and running an installer.

Looks like I'm sticking with 5.1 for now. Maybe next time I'll consider the HCL as gospel. There's a reason my motherboard's not certified for 5.5!

 

[_Gea]

There is a bug with 5.5 and e1000 vnics.
Try vmxnet3 vnics or disable LSO on e1000.
add the following lines to /kernel/drv/e1000g.conf (Illumos bases systems, reboot required)

#Disable TCP segmentation offload and LSO (napp-it all-in-one)
tx_hcksum_enable=0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0;
lso_enable=0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0;


You should also not update your VMs
Sata passthrough is not supported (I have no problem with LSI HBA)

Lets hope for 5.5.1

 

[4saken]

As far as your 'B' issue, we only had that when we upgraded the HW level of the VMs to the latest, but if we left them alone they were still manageable via the C# client. Fortunately we have had very few issues, biggest one with legacy *nix VMs built with an old lab manager 4 yrs ago, with 5.5. Been forcing myself to use the web client since they are forcing that direction on us regardless. But i feel you.

 

[TeeJayHoward]

As far as your 'B' issue, we only had that when we upgraded the HW level of the VMs to the latest, but if we left them alone they were still manageable via the C# client.

I created a VM with the web client, and was unable to change anything about it with the C# client. Okay, I get that. I deleted that VM and created another one with the C# client. After I was done creating it, I added the controller as a passthrough device, and tried to reserve all memory. I was informed that I couldn't do that, since the vCSA was controlling that VM. I turned off the vCSA VM. I still couldn't modify it. So incredibly frustrating.

And this morning, my NAS decides to go belly-up. It's just not my week.

 

[dasaint]

There is a bug with 5.5 and e1000 vnics.
Try vmxnet3 vnics or disable LSO on e1000.
add the following lines to /kernel/drv/e1000g.conf (Illumos bases systems, reboot required)

#Disable TCP segmentation offload and LSO (napp-it all-in-one)
tx_hcksum_enable=0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0;
lso_enable=0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0;

Hey GEA this never fixed the issue for me, i went to VMXNet3's and the problem went away.

 

[dasaint]

I the web client.

Which, by the way, who's genius idea was THAT hunk of junk?

A) It creates VMs only usable by the web client.
B) It takes over control of all VMs on that host so that you can't modify anything EXCEPT through the web client.
C) It does B even if the vCenter Server Appliance is turned off. The only way to fix it is to reboot the host.
D) It utilizes 4GB+ RAM on the resource-limited server instead of 65MB of RAM on a client.
E) It takes FOREVER to do anything. Every time I click a button, I'm forced to wait several seconds for the page to update. (And this when it's the only VM running!)
F) You can't open the console unless the VM is turned on - So much for hitting keys early in the boot process!

I am VERY thankful I decided to install to a different USB stick instead of overwriting my 5.1 install. Fixing my backup server was as simple as plugging in the old stick, creating a new VM, passing through the controller, and running an installer.

Looks like I'm sticking with 5.1 for now. Maybe next time I'll consider the HCL as gospel. There's a reason my motherboard's not certified for 5.5!

Hey TeeJayHoward,

I have provided some answers to your comments just so you would know there are options to fix some of the issues you have (it is known HW10 is landlocked to the Webclient and there is a KB that talks about that)

A) With HW10 Yes it is limited to the Web Client anything HW9 or less is still manageable by the C# Client. When Creating a VM theres a specific section on "Select Compatibility" if you select vSphere 5.5 you are selecting HW10.
B) See (A) For Answer
C) With Solaris i found VMXNet3 is the best NIC To use lately, There is a nice guide on OmniOS if you are interested in configuring a VM For Solaris. Napp-It also has the guide.
D) ESXi/vSphere 5.5 Required more memory to add additional features like vSAN and other functionalities to it.
E) How much memory did you allocate to the vCenter?
F) What keys do you want to hit? Bios? Boot to NIC?

Is the Controller missing from the Host or is it that it will not passthrough? If its missing from being visible i have heard of some VIBs existing that might be added to resolve this issue. Then it will be easier to passthrough to the VMs...

Hope this information is useful

 

[MrGuvernment]

I thought there really was no real benefit to passing through cards direct to VM's as the gain was very minimal vs the potential problems it can cause...

 

[TeeJayHoward]

Is the Controller missing from the Host or is it that it will not passthrough? If its missing from being visible i have heard of some VIBs existing that might be added to resolve this issue. Then it will be easier to passthrough to the VMs...

The controller is integrated into the host's motherboard. The host BIOS can see the drives.

With ESXi 5.1 installed and the controller passed through to the VM, the VM can see the controller and the drives. Everything works beautifully.

With ESXi 5.5 installed and the controller passed through to the VM, the VM can see the controller, but not the drives. In a Windows VM, the controller shows a code 10. In a Solaris VM, lspci lists the controller, but echo|format doesn't show any drives attached.

I thought there really was no real benefit to passing through cards direct to VM's as the gain was very minimal vs the potential problems it can cause...

SATA RDMs aren't supported in ESXi. Sure, you can do it, but you shouldn't. The only recommended method I know of to send a local drive to a VM is to pass through the whole controller.

 

[danswartz]

" I thought there really was no real benefit to passing through cards direct to VM's as the gain was very minimal vs the potential problems it can cause... "

The performance difference may be minimal, but if you are doing something like a storage appliance (especially ZFS based), being able to hot-plug drives, access SMART info, etc, is very helpful.

 

[MrGuvernment]

Good to know, i would of hoped by now ESXi could handle that directly via the vm layer.

I do know it can read most raid cards and give drive info and array info.. at least with some of the boxes i have it on with LSI cards.

 

[danswartz]

You can fake some of it, but real hot plug and such? Not that I am aware of...

 

[lopoetve]

Good to know, i would of hoped by now ESXi could handle that directly via the vm layer.

I do know it can read most raid cards and give drive info and array info.. at least with some of the boxes i have it on with LSI cards.

All that works fine if you're using the card to create a datastore assuming the motherboard supports the sensors as well. The problem is, you want real ZFS (for obvious reasons) with direct access, and that's impossible on a vmfs datastore.

 

[tbg]

5.5 also broke passthrough of the 6 port onboard sata on my 2 supermicro boads, x8sil and x9srl. I pass them to solaris vms and noticed that the controller does seem to get passed through but somthing gets messed up, like the AHCI version in 5.5 shows all fs, and the os can't see the drives.

in 5.5
Jan 25 15:33:07 RS1 ahci: [ID 405770 kern.info] NOTICE: ahci1: hba AHCI version = ffff.fff
Jan 25 15:33:07 RS1 pcplusmp: [ID 272826 kern.info] pcplusmp: pciclass,010601 (ahci) instance 1 irq 0x20 vector 0x45 ioapic 0xff intin 0xff is bound to cpu 0

in 5.1 u1
Jan 26 01:19:46 RS2 ahci: [ID 405770 kern.info] NOTICE: ahci0: hba AHCI version = 1.30
Jan 26 01:19:46 RS2 pcplusmp: [ID 272826 kern.info] pcplusmp: pciclass,010601 (ahci) instance 0 irq 0x1b vector 0x42 ioapic 0xff intin 0xff is bound to cpu 0

I'm hoping it will get fixed when they update 5.5, but as they took a bunch of sata drivers out in 5.5 who knows.

 

[Meanee]

Something I discovered that if you cannot edit settings of VM through vSphere client (non-web), you can still manage it through VMWare Workstation 10. Just connect to your vCenter through it.