![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Hi all
Just wondering if anyone has played around with IPoIB in ESXi 5.1 and if so what kind of speeds are you seeing in terms of data transfer using the IB fabric? (I understand the Mellanox OFED drivers don't work with 5.5 which is why I haven't upgraded yet
)
Here's my basic setup (soon to be upgraded again, just waiting for the typical end of month payday!):
I have purchased the following (I reliase its not cutting edge, but early next year I will be upgrading to a QDR / 40Gbps Mellanox IB switch with built-in SM) assuming I can get this working the way I expect:
1 x Voltaire GridDirector ISR 9024D (not the M model)
2 x MHGH28-XTC (Rev X1) HCA cards - I flashed these to firmware version 2.7000
2 x CX4 cables
2 x VMware ESXi custom systems
2 x Intel 335 SSDs (500MB/s each) - in 2 weeks this will become 4 x Intel 335 SSDs (providing theoretical 2Gbps ish IO in RAID-0)
Ok, I have installed the relevant drivers - for sake of a simple guide (which can be corrected if you think I have missed / done something wrong) here is what I did:
-------------------------
[ INFINIBAND ]
1. Install the Mellanox OFED drivers
esxcli system module paramters set -m=mlx4_core -p=mtu_4k=1
esxcli software vib install -d /tmp/mlx4_en-mlnx-1.6.1.2-offline_bundle-471530.zip no-sig-check
esxcli software vib install -d /tmp/MLNX-OFED-ESX-1.8.2.0.zip
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: Mellanox_bootbank_net-ib-cm_1.8.2.0-1OEM.500.0.0.472560, Mellanox_bootbank_net-ib-core_1.8.2.0-1OEM.500.0.0.472560, Mellanox_bootbank_net-ib-ipoib_1.8.2.0-1OEM.500.0.0.472560, Mellanox_bootbank_net-ib-mad_1.8.2.0-1OEM.500.0.0.472560, Mellanox_bootbank_net-ib-sa_1.8.2.0-1OEM.500.0.0.472560, Mellanox_bootbank_net-ib-umad_1.8.2.0-1OEM.500.0.0.472560, Mellanox_bootbank_net-mlx4-core_1.8.2.0-1OEM.500.0.0.472560, Mellanox_bootbank_net-mlx4-ib_1.8.2.0-1OEM.500.0.0.472560, Mellanox_bootbank_scsi-ib-srp_1.8.2.0-1OEM.500.0.0.472560
VIBs Removed:
VIBs Skipped:
esxcli software acceptance set --level=CommunitySupported
esxcli software vib install -v /tmp/ib-opensm-3.3.16.x86_64.vib --no-sig-check
2. Reboot
3. Fix MTU and partitions.conf
vi /tmp/partitions.conf
Default=0x7fff,ipoib,mtu=5:ALL=full;
cp partitions.conf /scratch/opensm/0x001a4bffff0c1399/
cp partitions.conf /scratch/opensm/0x001a4bffff0c139a/
4. Flashed both HCA cards to firmware 2.7000
5.Created a virtual network in ESXi using one port on the HCA each (per ESXi system) - ESXi recognises this vnic as up and 20Gbps
6. TRIED to set the MTU > 2k but failed, won't go higher than 2k in the vswitch.
7. Created 2 x WIN7 systems each with 2x4GHz vCPUs, 8GB RAM, 1 x SSD based HDD (theoretical 500MB/s or slightly less IO - no other VM using this SSD datastore) and configured NICs using IP on the IPoIB vswitch same subnet, ping works etc
8. Copied a 3.6GB ISO from WIN701 to WIN702 - 289Mbps (15secs) - thats fast but I was expecting more throughput
9. Created a 4GB RAM disk on each system
10. Re-copied the above file, result: 360MB/sec
---------------------------------
I was expecting much quicker than this copy rates, especially via RAMdisk
I then tested with a couple of CentOS Linux VM's with the MTU set on the VMXNET3 NIC:
[root@ib-lnx-01 ~]# rsync -av --progress /tmp/SM-6.3.2.0.632023-e50-00_OVF10.ova [email protected]:/tmp
[email protected]'s password:
sending incremental file list
SM-6.3.2.0.632023-e50-00_OVF10.ova
3370676736 100% 217.62MB/s 0:00:14 (xfer#1, to-check=0/1)
^[[28~
sent 232340 bytes received 464511 bytes 16019.56 bytes/sec
total size is 3370676736 speedup is 4837.01
Still nowhere near what I'd expect, wondering if its simply IPoIB itself (Mellanox are releasing V2.0 of the drivers for ESXi very soon.... apparently) or its the old tech HCA cards, anyone else got better results?
Just wondering if anyone has played around with IPoIB in ESXi 5.1 and if so what kind of speeds are you seeing in terms of data transfer using the IB fabric? (I understand the Mellanox OFED drivers don't work with 5.5 which is why I haven't upgraded yet
)Here's my basic setup (soon to be upgraded again, just waiting for the typical end of month payday!):
I have purchased the following (I reliase its not cutting edge, but early next year I will be upgrading to a QDR / 40Gbps Mellanox IB switch with built-in SM) assuming I can get this working the way I expect:
1 x Voltaire GridDirector ISR 9024D (not the M model)
2 x MHGH28-XTC (Rev X1) HCA cards - I flashed these to firmware version 2.7000
2 x CX4 cables
2 x VMware ESXi custom systems
2 x Intel 335 SSDs (500MB/s each) - in 2 weeks this will become 4 x Intel 335 SSDs (providing theoretical 2Gbps ish IO in RAID-0)
Ok, I have installed the relevant drivers - for sake of a simple guide (which can be corrected if you think I have missed / done something wrong) here is what I did:
-------------------------
[ INFINIBAND ]
1. Install the Mellanox OFED drivers
esxcli system module paramters set -m=mlx4_core -p=mtu_4k=1
esxcli software vib install -d /tmp/mlx4_en-mlnx-1.6.1.2-offline_bundle-471530.zip no-sig-check
esxcli software vib install -d /tmp/MLNX-OFED-ESX-1.8.2.0.zip
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: Mellanox_bootbank_net-ib-cm_1.8.2.0-1OEM.500.0.0.472560, Mellanox_bootbank_net-ib-core_1.8.2.0-1OEM.500.0.0.472560, Mellanox_bootbank_net-ib-ipoib_1.8.2.0-1OEM.500.0.0.472560, Mellanox_bootbank_net-ib-mad_1.8.2.0-1OEM.500.0.0.472560, Mellanox_bootbank_net-ib-sa_1.8.2.0-1OEM.500.0.0.472560, Mellanox_bootbank_net-ib-umad_1.8.2.0-1OEM.500.0.0.472560, Mellanox_bootbank_net-mlx4-core_1.8.2.0-1OEM.500.0.0.472560, Mellanox_bootbank_net-mlx4-ib_1.8.2.0-1OEM.500.0.0.472560, Mellanox_bootbank_scsi-ib-srp_1.8.2.0-1OEM.500.0.0.472560
VIBs Removed:
VIBs Skipped:
esxcli software acceptance set --level=CommunitySupported
esxcli software vib install -v /tmp/ib-opensm-3.3.16.x86_64.vib --no-sig-check
2. Reboot
3. Fix MTU and partitions.conf
vi /tmp/partitions.conf
Default=0x7fff,ipoib,mtu=5:ALL=full;
cp partitions.conf /scratch/opensm/0x001a4bffff0c1399/
cp partitions.conf /scratch/opensm/0x001a4bffff0c139a/
4. Flashed both HCA cards to firmware 2.7000
5.Created a virtual network in ESXi using one port on the HCA each (per ESXi system) - ESXi recognises this vnic as up and 20Gbps
6. TRIED to set the MTU > 2k but failed, won't go higher than 2k in the vswitch.
7. Created 2 x WIN7 systems each with 2x4GHz vCPUs, 8GB RAM, 1 x SSD based HDD (theoretical 500MB/s or slightly less IO - no other VM using this SSD datastore) and configured NICs using IP on the IPoIB vswitch same subnet, ping works etc
8. Copied a 3.6GB ISO from WIN701 to WIN702 - 289Mbps (15secs) - thats fast but I was expecting more throughput
9. Created a 4GB RAM disk on each system
10. Re-copied the above file, result: 360MB/sec
---------------------------------
I was expecting much quicker than this copy rates, especially via RAMdisk
I then tested with a couple of CentOS Linux VM's with the MTU set on the VMXNET3 NIC:
[root@ib-lnx-01 ~]# rsync -av --progress /tmp/SM-6.3.2.0.632023-e50-00_OVF10.ova [email protected]:/tmp
[email protected]'s password:
sending incremental file list
SM-6.3.2.0.632023-e50-00_OVF10.ova
3370676736 100% 217.62MB/s 0:00:14 (xfer#1, to-check=0/1)
^[[28~
sent 232340 bytes received 464511 bytes 16019.56 bytes/sec
total size is 3370676736 speedup is 4837.01
Still nowhere near what I'd expect, wondering if its simply IPoIB itself (Mellanox are releasing V2.0 of the drivers for ESXi very soon.... apparently) or its the old tech HCA cards, anyone else got better results?
