Customization gone haywire

[Paladin21]

I've got a cluster of ESXi 5.1 hosts (fully updated, etc) that have been running fine for over a year without issues. This semester, we're having a class that needs their machines to be on a private network to guarantee that they don't break stuff on the main network. I set up a dvswitch for internal only, and added a bunch of VMs to it. So far, so good. The professor teaching the class asked that they get network access, so I set up a router VM, bridged between the normal dvswitch and the new internal only one.

Here's where the fun stuff starts. The machines on the internal switch won't get an IP address from the router unless they're on the same physical host. If they're on a different one, they won't talk to the DHCP server. If I migrate one to the host where the router is, they get an address, and then the customization wizard kicks off and redoes the deployment (for the fourth or fifth time on some of these I've been testing). When they come back, they work fine...until DRS moves them to a different physical host. When that happens, customization kicks off again.

At this point I'm not sure why customization keeps kicking off (though I know I can manually edit it out of the configuration, there's ~200 VMs so I need to figure out why it's happening. I'm also not sure why the internal dvswitch doesn't relay across hosts. Any help would be appreciated. I haven't done much with the dvswitch functionality at all, past setting up the original one for normal access.

 

[peanuthead]

Anyway to provide a network diagram (without breaking any NDAs) to help wrap our brain around what you are working with?

 

[Paladin21]

I'll try to get something later, but I'd need approval to release anything like that. Basically there's one connection from our internal network to the physical switch that handles VM traffic. All ESXi hosts have two connections to this switch. There's a separate switch for NFS traffic for the datastores, with different connectivity. That's the physical side.

Configuration wise, each ESXi host has both pNICs bound to a dvSwitch that runs externally-accessible VMs. Previously, this is all we had and it was fine. What we've added is an additional dvSwitch that is host only (i.e. no attached pNICs/uplinks) for internal-only testing stuff. I added a router that has one nic on the external network, and one on the internal.

Question: I haven't found anything in the documentation, but does a dvSwitch need a physical uplink to relay broadcast traffic? I can deal with the configurator going nuts, I just don't understand why the internal dvSwitch won't relay broadcast DHCP.

 

[malingjc]

If I'm processing it properly, he has a DVS that has no physical uplink ports. Since there is no physical uplink ports, there is no physical medium between hosts. That means that connectivity to the 'router' VM doesn't exist if it's on another host.

Solution would probably be setting up a port group on the main interfaces but on a separate VLAN, which would likely be private enough for your purposes.

I'm not sure on the customization restart though - that's an odd one.

 

[Paladin21]

So I do need a physical link to bounce traffic through; that solves that issue. I had thought that the dvSwitches would work more like a "super-sized" version of the normal internal network switches, in that they would communicate directly from host to host instead of going outside the switch. I guess we're not quite there yet on that one.

On the configurator, it looks like it's kicking off any time a VM switches networks. Out of curiosity, I dropped a VM from the internal network into the one that's externally accessible, and it kicked the process again. I'm assuming that it works in reverse as well, but I don't have a good candidate to kick offline.