ESXi test lab: bad networking?

[exchange keys]

I built this test lab so that I can do software upgrades and test out new applications. Unfortunately, it seems like it is very slow, and whenever I ping the 5 virtual machines (either by IP address or DNS name), it times out. So, packets are dropping. Also, the virtual machines say "unidentified network" and I can't switch them back to Private networks. I ended up changing the Network List Manager Policies (secpol.msc) where "Unidentified Networks" are private to have a workaround. DNS names resolve, but like I said previously, the echo icmp requests time out every so often.

I have two resource pools:

Resource Pool A: 4 virtual machines (Win2k8 R2 SP1, 1 vCPU, E1000 NIC, 40 GB HDD)
Reservation - MB: 2234
Limit - MB: Unlimited
Shares: Normal
Shares Value: 163840
% Shares: 50
Type: Expandable

NOTE: One of the VMs is for MS SQL Server 2008 R2 SP1. All VMs have .NET Framework 3.5 SP1 and IIS 7 installed. One VM has a shared folder.

Resource Pool B: 1 virtual machine (Win2k8 R2 SP1, 2 vCPU, E1000 NIC, 20 GB HDD)
Reservation - MB: 0
Limit - MB: Unlimited
Shares: Normal
Shares Value: 163840
% Shares: 50
Type: Expandable

NOTE: This VM acts as my DHCP and DNS server. Just once instance of both services.

See image below for port group/network layout:

http://imageshack.us/photo/my-images/829/esxinet.png/

I'm also using a PowerConnect 3324 for a simple L2 switch. All ports are one VLAN. The two physical ports are connected to the PowerEdgep 2950's NICs which are NIC Teamed.

Summary:
---------------
Physical Host: Dell PowerEdge 2950
Processors: 4 Intel Xeon CPU x 1.862 GHz
Architecture: 64-bit
Hyperthreading: Inactive (I dunno why it's not turned on...)
Number of NICs: 2
Physical Memory: 4 Crucial 1 GB 140-pin 128MX72 DDR2 PC2-5 memory sticks
Maximum RAM = 32 GB
Minimum RAM = 512 MB
Memory Module Sockets = 8 (240 pins) slots
Maximum Memory per Module Socket = 4 GB
Memory Architecture = 533 or 667 (when available) MHz Full Buffered DIMMs (FBD)

Network Adapters:
---------------------------
Make/Model: Broadcom NetXtreme II BCM5708 1000Base-T
NICs: vmnic1, vmnic2
Speed: 100 Full Duplex
Configured: Negotiate
vSwitch: vSwitch0
Observed IP ranges: 172.16.42.1-172.16.42.127




I think I need more memory regardless, and maybe that is also why routing is so inconsistent. However, I don't know enough about ESXi/VMware networking to know for sure.

Troubleshooting Steps:

- Turned off all VMs and then restarted the ESXi host
- Turned on two VMs (one from each resource pool) and noticed the speed was faster
- pinging localhost was consistent (no timed outs)
- pinging the two servers by IP address seemed consistent
- Turned on other VMs and remoting via console seemed slower
- pinging localhost was still consistent (no timed out)
- pinging the five servers by IP address dropped packets
- pinging the five servers by DNS/FQDN dropped packets

Eventually, the VM froze and I couldn't use console. Tried RDP, and that couldn't load personal setting (dropped RDP session).

Some aid would be cool. Thank you.

 

[NetJunkie]

You say the switch is set for NIC teaming. What happens if you unteam them? By default, vSphere is going to hash NIC teams by Virtual Port ID, which is basically the same as by MAC address. You don't need to do anything to the switch for that to work. If you want to do, say, hashing by IP you'll need the switch to be configured for that.

I'm wondering if your hashing types are mismatched and that's why connectivity is questionable.

 

[NetJunkie]

Also, don't do memory reservations unless absolutely necessary. A bad idea in almost all cases. Assign the VMs what they should have, but don't reserve it. Use shares. If you do that then another VM can't use that RAM even if the first VM isn't doing anything with it.

How much RAM are you giving to these VMs? I see you only have 4GB total in the box. That's not much at all for 5 VMs.

 

[exchange keys]

Each VM is 1024 GB, so I don't have enough physical RAM (4 GBs).

I also noticed that my Network Adapters changed Observed IP Ranges. They are now: 168.0.0.1-175.255.255.254

The Network Adapters are configured to "autonegotiate" as well...

I'll try taking them off NIC Team and see if that does anything.

 

[lopoetve]

Also, don't do memory reservations unless absolutely necessary. A bad idea in almost all cases. Assign the VMs what they should have, but don't reserve it. Use shares. If you do that then another VM can't use that RAM even if the first VM isn't doing anything with it.

How much RAM are you giving to these VMs? I see you only have 4GB total in the box. That's not much at all for 5 VMs.

You know, you'd be surprised what you can get away with

My current record is 22GB allocated on an 8GB server, including a 15 desktop view deployment, VC 5.0VA, VC5.0 2k8 box, Linux LAMP server, 2 VSAs replicating, vRanger test box running constant backups, and a spare couple of 2k3 boxes (thinapp repo, sharepoint)...



Yes I'm insane, yes the view deployment was fully functional and performing fine (5 desktops deployed at the time and running with users, 3 on standby, windows 7 64bit), and yes it ran just happy as a clam. Good tuning in place though, and some really well configured resource pools.

 

[exchange keys]

Thank you for the information. I ended up removing the 5th VM (WDC) and Resource Pool B, revert back the CPU/Memory reservations, set both network adapters (vmnic0 and vmnic1) to auto-negotiate, and made the vSwitch0 only 24 ports instead of 56. I restarting the ESXi host two times (my vmnics became inactive and wouldn't send any traffic out to the physical ports on the PowerConnect 3324), and now things seem to be peachy. This has been most educational

Although, I am not sure if I am truly NIC teaming. I SSH'd into the ESXi host and did "esxtop" (then 'N') and noticed that vmnic0 and vmnic1 weren't always sending traffic. If I look through the VIC, it looks like the NICs are teaming. How do I know for sure?

 

[NetJunkie]

If they are in the same vSwitch and both set to Active they are teaming. You have to understand that teaming/port-channel/etherchannel (whatever you want to call it) is not true load balance. A connection to or from a VM will only flow over one NIC...there is no way to balance a single connection across multiple NICs due to how connections are hashed. That's why you normally won't see all NICs in use until you have a number of clients hitting VM servers.