Vmotion failed at 10% two way link

B

bain8848

n00b
Joined
Jul 26, 2011
Messages
14
Hi,
I'm getting in trouble with vmotion at 10%.The success percent is about 67%.The environment is : One vcenter server,two ESX server with HA, three cisco 2960 switchs.
Vcenter connected switch A, which connected switch B and switch C,all of these ports belong to vlan 2. There are two ESX server ,each server have four Nic ports.Two ports are service console ,which belong to vlan 2,and connected switch B and switch C. The other two ports are vmkernel ports,which belong to vlan 3, and connected switch B and switch C.All of these ports on switch are access mode.Only the two ports which connected Swtich B and switch C are trunk mode.
The vmotion success percent is about 67%, I have checked vmkping on ESX server and both of them are ok. And there are no error logs on ESX vmkernel log when vmotion failed.
But when I disconnect cabel between the switch A and switch C , or switch A and switch B,meaning one way.The success percent grows to 90% percent.
I have searched it at Vmware website, the answer is bollow:
http://www.vmwarewolf.com/vmotion-fails-at-10-percent/
And I have check all of these situation,but I didn't get any result.
I have no idea about it.Is there any special configuration on switch?
 
Running spanning-tree? Have a loop somewhere that goes away when you disconnect the cable? If you can plug the hosts in to a single switch and disconnect the others is it 100%?
 
Hi NetJunkie,
Thanks.I think there are no loop.The spanning-tree have blocked the link between switch A and switch C. Ping is ok always.
The following is the information for "show spanning vlan 2":

XYUN-L3SW04#sh sp vlan 2

VLAN0002
Spanning tree enabled protocol ieee
Root ID Priority 20482
Address e804.62c0.2800
Cost 7
Port 72 (Port-channel3)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 40962 (priority 40960 sys-id-ext 2)
Address b8be.bf4e.3880
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi0/1 Desg FWD 4 128.1 P2p
Gi0/5 Desg FWD 4 128.5 P2p
Gi0/6 Desg FWD 4 128.6 P2p
Gi0/8 Altn BLK 10 160.8 P2p
Po1 Desg FWD 3 128.56 P2p
Po2 Desg FWD 3 128.64 P2p
Po3 Root FWD 3 128.72 P2p


XYUN-L3SW04#sh sp vlan 2 de
XYUN-L3SW04#sh sp vlan 2 detail

VLAN0002 is executing the ieee compatible Spanning Tree protocol
Bridge Identifier has priority 40960, sysid 2, address b8be.bf4e.3880
Configured hello time 2, max age 20, forward delay 15
Current root has priority 20482, address e804.62c0.2800
Root port is 72 (Port-channel3), cost of root path is 7
Topology change flag not set, detected flag not set
Number of topology changes 8 last change occurred 03:58:47 ago
from Port-channel3
Times: hold 1, topology change 35, notification 2
hello 2, max age 20, forward delay 15
Timers: hello 0, topology change 0, notification 0, aging 300

Port 1 (GigabitEthernet0/1) of VLAN0002 is designated forwarding
Port path cost 4, Port priority 128, Port Identifier 128.1.
Designated root has priority 20482, address e804.62c0.2800
Designated bridge has priority 40962, address b8be.bf4e.3880
Designated port id is 128.1, designated path cost 7
Timers: message age 0, forward delay 0, hold 0
Number of transitions to forwarding state: 1
Link type is point-to-point by default
BPDU: sent 7350, received 0

Port 5 (GigabitEthernet0/5) of VLAN0002 is designated forwarding
Port path cost 4, Port priority 128, Port Identifier 128.5.
Designated root has priority 20482, address e804.62c0.2800
Designated bridge has priority 40962, address b8be.bf4e.3880
Designated port id is 128.5, designated path cost 7
Timers: message age 0, forward delay 0, hold 0
Number of transitions to forwarding state: 1
Link type is point-to-point by default
BPDU: sent 7351, received 0

Port 6 (GigabitEthernet0/6) of VLAN0002 is designated forwarding
Port path cost 4, Port priority 128, Port Identifier 128.6.
Designated root has priority 20482, address e804.62c0.2800
Designated bridge has priority 40962, address b8be.bf4e.3880
Designated port id is 128.6, designated path cost 7
Timers: message age 0, forward delay 0, hold 0
Number of transitions to forwarding state: 1
Link type is point-to-point by default
BPDU: sent 7351, received 0

Port 8 (GigabitEthernet0/8) of VLAN0002 is alternate blocking
Port path cost 10, Port priority 160, Port Identifier 160.8.
Designated root has priority 20482, address e804.62c0.2800
Designated bridge has priority 28674, address e804.62ca.2000
Designated port id is 128.16, designated path cost 3
Timers: message age 3, forward delay 0, hold 0
Number of transitions to forwarding state: 1
Link type is point-to-point by default
BPDU: sent 6, received 7347

Port 56 (Port-channel1) of VLAN0002 is designated forwarding
Port path cost 3, Port priority 128, Port Identifier 128.56.
Designated root has priority 20482, address e804.62c0.2800
Designated bridge has priority 40962, address b8be.bf4e.3880
Designated port id is 128.56, designated path cost 7
Timers: message age 0, forward delay 0, hold 0
Number of transitions to forwarding state: 1
Link type is point-to-point by default
BPDU: sent 7351, received 0

Port 64 (Port-channel2) of VLAN0002 is designated forwarding
Port path cost 3, Port priority 128, Port Identifier 128.64.
Designated root has priority 20482, address e804.62c0.2800
Designated bridge has priority 40962, address b8be.bf4e.3880
Designated port id is 128.64, designated path cost 7
Timers: message age 0, forward delay 0, hold 0
Number of transitions to forwarding state: 1
Link type is point-to-point by default
BPDU: sent 7351, received 0

Port 72 (Port-channel3) of VLAN0002 is root forwarding
Port path cost 3, Port priority 128, Port Identifier 128.72.
Designated root has priority 20482, address e804.62c0.2800
Designated bridge has priority 36866, address b8be.bf4e.6280
Designated port id is 128.72, designated path cost 4
Timers: message age 2, forward delay 0, hold 0
Number of transitions to forwarding state: 1
Link type is point-to-point by default
BPDU: sent 3, received 7165
 
The other switch is:
XYUN-L3SW03#sh sp vlan 2

VLAN0002
Spanning tree enabled protocol ieee
Root ID Priority 20482
Address e804.62c0.2800
Cost 4
Port 8 (GigabitEthernet0/8)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 36866 (priority 36864 sys-id-ext 2)
Address b8be.bf4e.6280
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi0/1 Desg FWD 4 128.1 P2p
Gi0/5 Desg FWD 4 128.5 P2p
Gi0/6 Desg FWD 4 128.6 P2p
Gi0/8 Root FWD 4 128.8 P2p
Po1 Desg FWD 3 128.56 P2p
Po2 Desg FWD 3 128.64 P2p
Po3 Desg FWD 3 128.72 P2p

This is root switch.
 
Has this always been the case or did the issue just arise? I would try the recommendation from an earlier post: place all the NIC's into a single switch and see what happens.

Also shouldn't the link between Switch A and C be trunked as well?
 
Last edited:
Remove your link between A & C and place VCenter machine in either B or C and see what happens.
 
Hi jenuster
Remove your link between A & C and place VCenter machine in either B or C and see what happens.
>>>>I'll test it after a while.

Has this always been the case or did the issue just arise? I would try the recommendation from an earlier post: place all the NIC's into a single switch and see what happens.
>>>>I have put them on one switch,including vcenter,one of esx console port and one of vmkernel port. All of the esx server are the same.But the test result are not good.

Also shouldn't the link between Switch A and C be trunked as well?
>>>>I have test it.The same result.
 
have these switches been used before for other things? I would also reset to default config and try everything on that one switch.
 
Hi AMD Gamer,
These are new switches.At first,I suspect it's a loop problem.But even I change the spanning-tree cost manually, the problem are still. But when I disconect the link between switch A and B, or switch A and C, it turn to better. So I think maybe it's the switch configuration problem.Or vmotion don't support my topology.
 
Hi
When I put the vcenter at switch B and disconnect the link between switch A&C,the success percent grows to 100%! But when I put vcenter at switch C and disconnect the link between switch A&C,the success percent fall to 57%!
The vcenter gateway live in switch A.
Does it relate with vcenter location?

And when there are two links, A-B and A-C,the success percent go to worse.
 
Switch A configuration:

!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname SwitchA
!
boot-start-marker
boot-end-marker
!
enable password cisco
!
no aaa new-model
system mtu routing 1500
ip subnet-zero
!
!
!
crypto pki trustpoint TP-self-signed-4217705472
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-4217705472
revocation-check none
rsakeypair TP-self-signed-4217705472
!
!
crypto pki certificate chain TP-self-signed-4217705472
certificate self-signed 01
3082023F 308201A8 A0030201 02020101 300D0609 2A864886 F70D0101 04050030
31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
69666963 6174652D 34323137 37303534 3732301E 170D3933 30333031 30303031
31345A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649
4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D34 32313737
30353437 3230819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281
8100A7DE F4ACED7B 245B7F0D 3DED2050 83978E86 6F15E957 60528827 25E8C3C6
0EFEDF8A 60A96056 807F67C7 5C8954DD DC6F2100 163F8937 1D556728 833735A5
57E358E3 DBF867DD 77062CFC A5442C4C 8DDCE3B6 2A109F72 B2AB911C 3B167F67
512BE8E1 8D041330 B0062970 40FECB74 BA8AB821 7D9CF42E D19D70C2 2D773AC2
37850203 010001A3 67306530 0F060355 1D130101 FF040530 030101FF 30120603
551D1104 0B300982 07537769 7463682E 301F0603 551D2304 18301680 14EA0B21
27BD3275 B405C0E0 DDBAAA46 57C09876 83301D06 03551D0E 04160414 EA0B2127
BD3275B4 05C0E0DD BAAA4657 C0987683 300D0609 2A864886 F70D0101 04050003
8181002B 00157C71 6F3364A9 E4CEC06B 5097C4A4 E5297F60 A92745EB 402F6ACB
546B8ED9 0CB2C84B D57676B9 3FB40009 1328FE79 8DFB7464 191D6FCB 784F256F
5966D220 5FB207DB 1355054A 99968F5E 55BEE14F F3F4BE99 D05879F6 619E848F
6224C552 FE08F18A AA528C65 D7D123E1 E28136BC F658BBBF 61A1E700 B431AF35 91F80C
quit
!
!
!
!
!
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
!
!
interface GigabitEthernet0/1
switchport access vlan 2
switchport mode access
!
interface GigabitEthernet0/2
switchport access vlan 2
switchport mode access
!
interface GigabitEthernet0/3
switchport access vlan 2
switchport mode access
!
interface GigabitEthernet0/4
switchport access vlan 2
switchport mode access
!
interface GigabitEthernet0/5
!
interface GigabitEthernet0/6
!
interface GigabitEthernet0/7
!
interface GigabitEthernet0/8
!
interface GigabitEthernet0/9
!
interface GigabitEthernet0/10
!
interface GigabitEthernet0/11
!
interface GigabitEthernet0/12
!
interface GigabitEthernet0/13
duplex full
!
interface GigabitEthernet0/14
duplex full
!
interface GigabitEthernet0/15
!
interface GigabitEthernet0/16
!
interface GigabitEthernet0/17
!
interface GigabitEthernet0/18
!
interface GigabitEthernet0/19
!
interface GigabitEthernet0/20
switchport access vlan 2
switchport mode access
!
interface GigabitEthernet0/21
!
interface GigabitEthernet0/22
!
interface GigabitEthernet0/23
!
interface GigabitEthernet0/24
!
interface Vlan1
no ip address
no ip route-cache
!
interface Vlan2
ip address 10.64.2.254 255.255.255.0
no ip route-cache
!
interface Vlan200
no ip address
no ip route-cache
!
ip http server
ip http secure-server
!
control-plane
!
!
line con 0
line vty 0 4
password cisco
login
line vty 5 15
login
!
!
monitor session 1 source interface Gi0/1 - 3
end

SwitchA#

10.64.2.254 is the Vcenter Gateway
Vlan 2 is the vcenter and service console vlan.
 
Your problem isn't vCenter or vSphere..it's your network. Post the configs for all 3 switches and exactly which ports connect to which other ports. Also, confirm you don't have a bad NIC in a server.
 
lopoetve said:
At the end of the Vmotion, the VM's MAC address exists in 2 places at once for up to a second.

Spanning Tree gets mighty confused by that :)
Click to expand...

Why? Spanning-tree doesn't track MAC addresses. Dupe MACs would not cause a topology change...though yes, put it in port-fast.
 
NetJunkie said:
Why? Spanning-tree doesn't track MAC addresses. Dupe MACs would not cause a topology change...though yes, put it in port-fast.
Click to expand...

dup macs would not cause a topology change, but if the switch thinks mac X is behind port A and not port B when in learning state, it won't forward the frames. Or am I misunderstanding STP? I freely admit I'm not that savvy about L2...
 
NetJunkie said:
Why? Spanning-tree doesn't track MAC addresses. Dupe MACs would not cause a topology change...though yes, put it in port-fast.
Click to expand...

It does use the switches MAC fro the bridge ID. But i think he meant to say that the switches get confused by the same mac addressee in two places.
 
AMD_Gamer said:
It does use the switches MAC fro the bridge ID. But i think he meant to say that the switches get confused by the same mac addressee in two places.
Click to expand...

I know what he meant. :) Two MACs on two ports, even on the same switch, will not cause spanning-tree to do anything. It doesn't work at that level.

Now, you should absolutely disable STP on any port going to a vSphere host. That I'm in absolute agreement with.
 
My understanding was that if a MAC address moves to a different port, it won't be able to be reached their until it ages out of the learning table. Does portfast accelerate that? If not, why does portfast prevent this?
 
danswartz said:
My understanding was that if a MAC address moves to a different port, it won't be able to be reached their until it ages out of the learning table. Does portfast accelerate that? If not, why does portfast prevent this?
Click to expand...

A MAC won't live on more than one port (or logical port in case of a channel) at a time. As soon as a switch sees that MAC show up on another port it is deleted from the MAC table of the old port.

EDIT: That's why in the case of a vMotion a host does a gratuitous ARP. That's a broadcast that tells every switch in the fabric that the MAC has moved. Without that if I moved a NIC from Switch A to Switch B and Switch A didn't see a frame from that host for a while any host on Switch A that tried to send to the host wouldn't be able to... The gratuitous ARP fixes that.
 
QHalo said:
http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=1003804

The resolution should explain it.
Click to expand...

Yes..that's on a failure type event which is why I say run portfast to vSphere hosts. That way a link bouncing doesn't cause the link to go through the listening/learning STP process for no reason (a vSphere host isn't another switch/bridge). But a NIC that doesn't get disconnected won't have an issue and shouldn't cause vMotion to fail. If a NIC is bouncing during a vMotion there is another problem that needs to be fixed.
 
NetJunkie said:
A MAC won't live on more than one port (or logical port in case of a channel) at a time. As soon as a switch sees that MAC show up on another port it is deleted from the MAC table of the old port.

EDIT: That's why in the case of a vMotion a host does a gratuitous ARP. That's a broadcast that tells every switch in the fabric that the MAC has moved. Without that if I moved a NIC from Switch A to Switch B and Switch A didn't see a frame from that host for a while any host on Switch A that tried to send to the host wouldn't be able to... The gratuitous ARP fixes that.
Click to expand...

I actually had to do something like this with a legacy enet teaming system some years back. e.g. when switching from one nic to the other, issue a gratuitous arp thru the new nic to cause the switch to update...
 
NetJunkie said:
A MAC won't live on more than one port (or logical port in case of a channel) at a time. As soon as a switch sees that MAC show up on another port it is deleted from the MAC table of the old port.

EDIT: That's why in the case of a vMotion a host does a gratuitous ARP. That's a broadcast that tells every switch in the fabric that the MAC has moved. Without that if I moved a NIC from Switch A to Switch B and Switch A didn't see a frame from that host for a while any host on Switch A that tried to send to the host wouldn't be able to... The gratuitous ARP fixes that.
Click to expand...

Our Senior network engineers would disagree. :) We always recommend enabling portfast on all vmotion networks for this (especially if they share with storage networks - it's fun watching NFS drop because you vmotion something).
 
NetJunkie said:
I know what he meant. :) Two MACs on two ports, even on the same switch, will not cause spanning-tree to do anything. It doesn't work at that level.

Now, you should absolutely disable STP on any port going to a vSphere host. That I'm in absolute agreement with.
Click to expand...

One mac on two ports, however, will make it check for a loop. With portfast off, no forwarding happens.

At least, that's what I've been told by people way above my pay grade, and I've seen it happen several times... In fact, I've been on the phone when engineering told a customer that precise thing. Enable portfast, see if the problem goes away.

All I know about networking is what I've been told. I'm a storage guy. Networking is voodoo shit. :p If it pings, I'm done ;)
 
lopoetve said:
One mac on two ports, however, will make it check for a loop. With portfast off, no forwarding happens.

At least, that's what I've been told by people way above my pay grade, and I've seen it happen several times... In fact, I've been on the phone when engineering told a customer that precise thing. Enable portfast, see if the problem goes away.

All I know about networking is what I've been told. I'm a storage guy. Networking is voodoo shit. :p If it pings, I'm done ;)
Click to expand...

Two MACs on a port will not cause STP to recalc anything. STP has no visibility in to the MAC table on a switch. It looks for BPDUs. If it sees one of those all of a sudden from a vSphere host it'll recalc. Or if it sees one come back around due to loop. But two MACs? Not a clue.

Point me to the Sr. Network guy. I'd like to have a conversation around that.
 
Just thought of something, what if they path cost changes somehow? maybe that is why they say to enable portfast?
 
AMD_Gamer said:
Just thought of something, what if they path cost changes somehow? maybe that is why they say to enable portfast?
Click to expand...

"Path Cost" wouldn't come in to play here as it is not going to change. If it changed up stream and caused a STP recalc that's fine....Putting the vSphere ports in portfast mode won't stop that. It will just keep it from going through the STP bring-up process just like it would for any STP recalc cause.
 
Yeah..i'm not sure where MAC addresses come into play using STP.

Portfast assumes the port is not part of a loop and puts it in forwarding. Most of the other port states are "skipped" with it enabled.
 
NetJunkie said:
Two MACs on a port will not cause STP to recalc anything. STP has no visibility in to the MAC table on a switch. It looks for BPDUs. If it sees one of those all of a sudden from a vSphere host it'll recalc. Or if it sees one come back around due to loop. But two MACs? Not a clue.

Point me to the Sr. Network guy. I'd like to have a conversation around that.
Click to expand...

Checking with him now to verify.
 
NetJunkie said:
Two MACs on a port will not cause STP to recalc anything. STP has no visibility in to the MAC table on a switch. It looks for BPDUs. If it sees one of those all of a sudden from a vSphere host it'll recalc. Or if it sees one come back around due to loop. But two MACs? Not a clue.

Point me to the Sr. Network guy. I'd like to have a conversation around that.
Click to expand...

can a host send out BPDU's? you would need BPDU Guard for that right?
 
Hosts won't send out BPDUs - working with someone now on how else we might end up with that. Gimme a bit.
 
AMD_Gamer said:
can a host send out BPDU's? you would need BPDU Guard for that right?
Click to expand...

Switches send out BPDUs (Bridge Protocol Data Unit). Hosts cannot send these out. BPDU guard is used on ports that have PortFast enabled to ensure that no switches downstream can be involved in the STP root bridge elections.
 
ok, so my cases (and the info) was pre-BPDUGuard and RSTP, when apparently the early versions of the protocol ~did~ look at the CAM table. It's still something we encounter apparently as well. No idea how old the code was or when that way (again, not a networking guy), but the folks I'm talking with know their shit so I tend to trust them.

I'll stand by my suggestion of enabling portfast and seeing if the problem goes away.
 
Hi dear all,
After I change the port to portfast which connect the EXS hosts,the vmotion success percent at 80%.And there are a vm host failed to swith HA,the other 4 vm hosts are ok.
 
You must log in or register to reply here.
Back
Top