• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Infiniband is making me crazy!

V

vFX

n00b
Joined
Sep 28, 2013
Messages
55
I have this "simple" setup

Node 1 "STORAGE"
OS: Ubuntu 14.04 (server)
IB: Mellanox ConnectX 2
Software installed: Mellanox OFED 2.3-2.0.0 (ubuntu14.04-x86_64)

Node 2 "HYPERV1"
OS: ESXi 5.5
IB: Mellanox ConnectX 2
Software installed: MLNX-OFED-ESX-1.8.2.0, mlx4_en-mlnx-1.6.1.2-471530, ib-opensm-3.3.16-64.x86_64

No IB Switch, the nodes are directly connected.

THE PROBLEM

The fisrt day everything worked well, I was able to export a NFS share from the storage node to the esxi and install a VM on it with a great performance.
The second day I had to reboot the storage node and after that I was no more able to make the connection work. No ibping, no ping.

Here are some details


STORAGE NODE

Code: 
[B]root@storage:~#ibv_devinfo[/B]
hca_id:	mlx4_0
	transport:			InfiniBand (0)
	fw_ver:				2.9.1000
	node_guid:			0002:c903:000d:1c08
	sys_image_guid:			0002:c903:000d:1c0b
	vendor_id:			0x02c9
	vendor_part_id:			26428
	hw_ver:				0xB0
	board_id:			MT_0D81120009
	phys_port_cnt:			2
		port:	1
			state:			PORT_DOWN (1)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		0
			port_lmc:		0x00
			link_layer:		InfiniBand

		port:	2
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			1
			port_lid:		2
			port_lmc:		0x00
			link_layer:		InfiniBand

Code: 
[B]root@storage:~# hca_self_test.ofed[/B]

---- Performing Adapter Device Self Test ----
Number of CAs Detected ................. 1
PCI Device Check ....................... PASS
Kernel Arch ............................ x86_64
Host Driver Version .................... MLNX_OFED_LINUX-2.3-2.0.0: 3.13.0-32-generic
Host Driver RPM Check .................. PASS
Firmware on CA #0 HCA .................. v2.9.1000
Firmware Check on CA #0 (HCA) .......... NA
    REASON: NO required fw version
Host Driver Initialization ............. PASS
Number of CA Ports Active .............. 1
Port State of Port #1 on CA #0 (HCA)..... DOWN (InfiniBand)
Port State of Port #2 on CA #0 (HCA)..... UP 4X QDR (InfiniBand)
Error Counter Check on CA #0 (HCA)...... PASS
Kernel Syslog Check .................... PASS
Node GUID on CA #0 (HCA) ............... 00:02:c9:03:00:0d:1c:08
------------------ DONE ---------------------


Code: 
[B]root@storage:~# ibstat[/B]
CA 'mlx4_0'
	CA type: MT26428
	Number of ports: 2
	Firmware version: 2.9.1000
	Hardware version: b0
	Node GUID: 0x0002c903000d1c08
	System image GUID: 0x0002c903000d1c0b
	Port 1:
		State: Down
		Physical state: Polling
		Rate: 10
		Base lid: 0
		LMC: 0
		SM lid: 0
		Capability mask: 0x02510868
		Port GUID: 0x0002c903000d1c09
		Link layer: InfiniBand
	Port 2:
		State: Active
		Physical state: LinkUp
		Rate: 40
		Base lid: 2
		LMC: 0
		SM lid: 1
		Capability mask: 0x02510868
		Port GUID: 0x0002c903000d1c0a
		Link layer: InfiniBand

Code: 
[B]root@storage:~# ibstatus[/B]
Infiniband device 'mlx4_0' port 1 status:
	default gid:	 fe80:0000:0000:0000:0002:c903:000d:1c09
	base lid:	 0x0
	sm lid:		 0x0
	state:		 1: DOWN
	phys state:	 2: Polling
	rate:		 10 Gb/sec (4X)
	link_layer:	 InfiniBand

Infiniband device 'mlx4_0' port 2 status:
	default gid:	 fe80:0000:0000:0000:0002:c903:000d:1c0a
	base lid:	 0x2
	sm lid:		 0x1
	state:		 4: ACTIVE
	phys state:	 5: LinkUp
	rate:		 40 Gb/sec (4X QDR)
	link_layer:	 InfiniBand

root@storage:~#

Code: 
[B]root@storage:~# ibhosts[/B]
Ca	: 0x0002c903000d246c ports 2 "hyperv1.home.lan HCA-1"
Ca	: 0x0002c903000d1c08 ports 2 "storage HCA-1"


Code: 
[B]root@storage:~# ifconfig ib1[/B]
ib1       Link encap:UNSPEC  HWaddr A0-00-02-20-FE-80-00-00-00-00-00-00-00-00-00-00  
          inet addr:10.0.1.11  Bcast:10.0.1.255  Mask:255.255.255.0
          inet6 addr: fe80::202:c903:d:1c0a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
          RX packets:48 errors:0 dropped:15 overruns:0 frame:0
          TX packets:41 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1024 
          RX bytes:2954 (2.9 KB)  TX bytes:3288 (3.2 KB)

Code: 
[B]root@storage:~# route[/B]
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         ibox.home.lan   0.0.0.0         UG    0      0        0 eth0
10.0.1.0        *               255.255.255.0   U     0      0        0 ib0
10.0.1.0        *               255.255.255.0   U     0      0        0 ib1
192.168.1.0     *               255.255.255.0   U     0      0        0 eth0

Code: 
[B]root@storage:~# ping 10.0.1.21[/B]
PING 10.0.1.21 (10.0.1.21) 56(84) bytes of data.
From 10.0.1.10 icmp_seq=1 Destination Host Unreachable
From 10.0.1.10 icmp_seq=2 Destination Host Unreachable
From 10.0.1.10 icmp_seq=3 Destination Host Unreachable




ESXi NODE


Code: 
[B]~ # /opt/opensm/bin/ibstat [/B]
CA 'mlx4_0'
	CA type: MT26428
	Number of ports: 2
	Firmware version: 2.9.1000
	Hardware version: b0
	Node GUID: 0x0002c903000d246c
	System image GUID: 0x0002c903000d246f
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 40
		Base lid: 1
		LMC: 0
		SM lid: 1
		Capability mask: 0x0251086a
		Port GUID: 0x0002c903000d246d
		Link layer: InfiniBand
	Port 2:
		State: Down
		Physical state: Polling
		Rate: 68
		Base lid: 0
		LMC: 0
		SM lid: 0
		Capability mask: 0x0251086a
		Port GUID: 0x0002c903000d246e
		Link layer: InfiniBand
~ #

Code: 
[B]~ # esxcli network ip interface ipv4 get[/B]
Name  IPv4 Address  IPv4 Netmask   IPv4 Broadcast  Address Type  DHCP DNS
----  ------------  -------------  --------------  ------------  --------
vmk0  192.168.1.10  255.255.255.0  192.168.1.255   STATIC           false
vmk1  10.0.1.21     255.255.255.0  10.0.1.255      STATIC           false
~ #


Code: 
[B]~ # esxcli network nic list[/B]
Name       PCI Device     Driver  Link  Speed  Duplex  MAC Address         MTU  Description                                                                   
---------  -------------  ------  ----  -----  ------  -----------------  ----  ------------------------------------------------------------------------------
vmnic0     0000:003:00.0  e1000e  Up     1000  Full    00:25:90:06:ca:16  1500  Intel Corporation 82574L Gigabit Network Connection                           
vmnic1     0000:004:00.0  e1000e  Down      0  Half    00:25:90:06:ca:17  1500  Intel Corporation 82574L Gigabit Network Connection                           
vmnic2     0000:007:00.0  e1000e  Up     1000  Full    00:15:17:d6:c6:26  1500  Intel Corporation 82571EB Gigabit Ethernet Controller                         
vmnic3     0000:007:00.1  e1000e  Up     1000  Full    00:15:17:d6:c6:27  1500  Intel Corporation 82571EB Gigabit Ethernet Controller                         
vmnic_ib0  0000:005:00.0          Up    40000  Full    00:02:c9:0d:24:6d  2044  Mellanox Technologies MT26428 [ConnectX VPI - 10GigE / IB QDR, PCIe 2.0 5GT/s]
vmnic_ib1  0000:005:00.0          Down      0  Half    00:02:c9:0d:24:6e  1500  Mellanox Technologies MT26428 [ConnectX VPI - 10GigE / IB QDR, PCIe 2.0 5GT/s]
~ #

Code: 
[B]~ # esxcli network vswitch standard list[/B]
vSwitch0
   Name: vSwitch0
   Class: etherswitch
   Num Ports: 1792
   Used Ports: 6
   Configured Ports: 128
   MTU: 1500
   CDP Status: listen
   Beacon Enabled: false
   Beacon Interval: 1
   Beacon Threshold: 3
   Beacon Required By: 
   Uplinks: vmnic3, vmnic2
   Portgroups: VM Network, Management Network

vSwitch1
   Name: vSwitch1
   Class: etherswitch
   Num Ports: 1792
   Used Ports: 4
   Configured Ports: 128
   MTU: 2044
   CDP Status: listen
   Beacon Enabled: false
   Beacon Interval: 1
   Beacon Threshold: 3
   Beacon Required By: 
   Uplinks: vmnic_ib0
   Portgroups: ib
~ #

Code: 
[B]~ # ping 10.0.1.11[/B]
PING 10.0.1.11 (10.0.1.11): 56 data bytes

--- 10.0.1.11 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
~ #



Additional info:
- opensm is not started on the storage node because it is running on the esxi one.
- the cards have both the last fw
- I was not able to use ibping both with -G and -L parameter
 
Install samba and try using that to share to the Hyper-v host. NFS and windows can be interesting.
 
Well, I see an MTU mismatch (esx = 2044, storage = 4096).

Was the only change otherwise a reboot of the storage system? Besides the ping tests, what else have you tried (services restart, anything)?
 
DeChache said:
Install samba and try using that to share to the Hyper-v host. NFS and windows can be interesting.
Click to expand...

HYPERV1 is the name of my VMware vSphere ESXi host, it's not a Microsoft HyperV server :D

Eulogy said:
Well, I see an MTU mismatch (esx = 2044, storage = 4096).

Was the only change otherwise a reboot of the storage system? Besides the ping tests, what else have you tried (services restart, anything)?
Click to expand...

the MTU mentioned in the devinfo output is 4096 but in ifconfig ib1 is 2044.
since I'm running IPoIB and I'm a ib newbie I'm not sure what exactly the 4096 in devinfo means, I suppose this is for "pure" ib traffic.

anyway I solved the problem moving the cable from the port 1 to the port 0!
now will be interesting to see what happen when I connect the second esxi node to the storage server.. :D
 
The reason for the loss of connectivity is most likely due to OpenSM. my question is the OFED driver you install in linux should also contain a OpenSM package... by happenstance did it start as a service? if so it would conflict with the other one (Raphael's hypervisor.fr - the ESXi Version you are using.) and potentially cause issues.

Check on this for me, as this sounds like what happened...
 
dasaint said:
The reason for the loss of connectivity is most likely due to OpenSM. my question is the OFED driver you install in linux should also contain a OpenSM package... by happenstance did it start as a service? if so it would conflict with the other one (Raphael's hypervisor.fr - the ESXi Version you are using.) and potentially cause issues.

Check on this for me, as this sounds like what happened...
Click to expand...

good point, at some point I thought the same thing and checked that opensm on the storage node was not running but the problem was something else, as said earlier


vFX said:
anyway I solved the problem moving the cable from the port 1 to the port 0!
now will be interesting to see what happen when I connect the second esxi node to the storage server.. :D
Click to expand...
 
vFX said:
HYPERV1 is the name of my VMware vSphere ESXi host, it's not a Microsoft HyperV server :D

the MTU mentioned in the devinfo output is 4096 but in ifconfig ib1 is 2044.
since I'm running IPoIB and I'm a ib newbie I'm not sure what exactly the 4096 in devinfo means, I suppose this is for "pure" ib traffic.

anyway I solved the problem moving the cable from the port 1 to the port 0!
now will be interesting to see what happen when I connect the second esxi node to the storage server.. :D
Click to expand...

I have had a similar issue with the MTU on ESXi not going above the 2044 even with the OFED driver if i tried to jump the VMKernel port above 2044 the esxi host would error in the logs about not being able to goto the 4092 range.. Even though the switch i have has the ability to go to the 4K MTU rating...

Something to also just mention that i had issues with. Jumping Ports on the back of the card is interesting i wonder if the LID wouldn't re-Register once you disconnected the port (made it in-active) did u try re-plugging it in after 5 minutes see if it started working?

Infiniband can defiantly drive one crazy for sure! I was fortunate enough to get my hands on a Connect-X3 Pro VPI cards + SX6012 Switch from to play in my lab... Having OpenSM managed switch really changes the game massively compared to using like what Raphael made. Before i got that switch/card set with my Connect-X1 cards and switch i considered creating a Box just to handle OpenSM (using a PCIe Card + Linux + OFED and OpenSM to manage the switch as a 3rd party) Once you go beyond 2 Machines it gets complex!
 
dasaint said:
I have had a similar issue with the MTU on ESXi not going above the 2044 even with the OFED driver if i tried to jump the VMKernel port above 2044 the esxi host would error in the logs about not being able to goto the 4092 range.. Even though the switch i have has the ability to go to the 4K MTU rating...

Something to also just mention that i had issues with. Jumping Ports on the back of the card is interesting i wonder if the LID wouldn't re-Register once you disconnected the port (made it in-active) did u try re-plugging it in after 5 minutes see if it started working?

Infiniband can defiantly drive one crazy for sure! I was fortunate enough to get my hands on a Connect-X3 Pro VPI cards + SX6012 Switch from to play in my lab... Having OpenSM managed switch really changes the game massively compared to using like what Raphael made. Before i got that switch/card set with my Connect-X1 cards and switch i considered creating a Box just to handle OpenSM (using a PCIe Card + Linux + OFED and OpenSM to manage the switch as a 3rd party) Once you go beyond 2 Machines it gets complex!
Click to expand...

to be completely honest I have still some problems to understand how opensm works.
I mean, the theory is "easy and obvious" but in practice I'm not sure how opensm works and what exactly I have configured.
maybe it's normal and I have just to practice more..

about the mtu I was pretty lucky, on my systems was easy to get 4092 working

Code: 
[B]~ # esxcli network nic list[/B]
Name       PCI Device     Driver  Link  Speed  Duplex  MAC Address         MTU  Description                                                                   
---------  -------------  ------  ----  -----  ------  -----------------  ----  ------------------------------------------------------------------------------
vmnic0     0000:003:00.0  e1000e  Up     1000  Full    00:25:90:06:ca:16  1500  Intel Corporation 82574L Gigabit Network Connection                           
vmnic1     0000:004:00.0  e1000e  Down      0  Half    00:25:90:06:ca:17  1500  Intel Corporation 82574L Gigabit Network Connection                           
vmnic2     0000:007:00.0  e1000e  Up     1000  Full    00:15:17:d6:c6:26  1500  Intel Corporation 82571EB Gigabit Ethernet Controller                         
vmnic3     0000:007:00.1  e1000e  Up     1000  Full    00:15:17:d6:c6:27  1500  Intel Corporation 82571EB Gigabit Ethernet Controller                         
[B]vmnic_ib0  0000:005:00.0          Up    40000  Full    00:02:c9:0d:24:6d  4092  Mellanox Technologies MT26428 [ConnectX VPI - 10GigE / IB QDR, PCIe 2.0 5GT/s]
[/B]vmnic_ib1  0000:005:00.0          Down      0  Half    00:02:c9:0d:24:6e  1500  Mellanox Technologies MT26428 [ConnectX VPI - 10GigE / IB QDR, PCIe 2.0 5GT/s]

Code: 
vSwitch1
   Name: vSwitch1
   Class: etherswitch
   Num Ports: 1792
   Used Ports: 4
   Configured Ports: 128
[B]   MTU: 4092
[/B]   CDP Status: listen
   Beacon Enabled: false
   Beacon Interval: 1
   Beacon Threshold: 3
   Beacon Required By: 
   Uplinks: vmnic_ib0
   Portgroups: ib-network
 
Did you Follow Raphael's stuff where creating the partitions.conf in the OpenSM scratch directories for both of your ports? This creates the Partitions or vLANS if u wanna call it in the Infiniband world. These partitions create the communication zone between objects.

Core concept is that it creates a container (partition) that houses interconnected devices if theres a switch it programs the switch to be the container if it cannot find a switch it containerizes locally with all the information needed to communicate between nodes.

its kinda like TCP as a container... without TCP u just got cables and plugins and devices no way to cross communicate... the TCP Stack when implemented it allows those separated devices communicate kind of like how a switch needs to understand the TCP Stack to keep track of ARP tables (LIDS in Infinbands case) and tells it how to route from A to B...

OpenSM on ESXi however instantiates per port so downside is if you connect both into the same container it sometimes can have wonky effects due to the active/passive nature of the ESXi OpenSM driver.

hope my rambles make sense... this is my understanding/perception of how IB works... might not be right tho... but thats how i interpret and use it and it works!
 
thank you for the explanation, it's very helpful have this stuff explained outside from the technical documents.

my plan is

esxi1+opensm (hca0,ib0) <------------------------> (hca0,ib0) storage
esxi2+opensm (hca0,ib0) <------------------------> (hca0,ib1) storage

so I should have no problems like managing two ports with one opensm instance.
of course with this config I will have HA and vMotion over ethernet 1G and not ib.
 
Glad it helps!

Your method would work, as long as you don't in ghostbusters terms... Cross the Streams... If you do end up with a switch then your issues would resolve but its not a cheap solution for sure... and lord forbit a switch with opensm on it is LOUD...

70 dBA is sometimes a little too much for a lab... at least for most... the other switch i had was silent however would require a 3rd party linux solution to program the switch. You just dont want openSM on a box that reboots often.
 
P.S. Selling off some of my older Infiniband Gear if anyone is interested!
 
You must log in or register to reply here.
Back
Top