10GBE question on cabling

L

luckylinux

Limp Gawd
Joined
Mar 19, 2012
Messages
225
I have bought a few of these single port devices to put in each ESXi box I own: Mellanox Connectx-2 MNPA19-XTR. I recently spotted a deal on a Mellanox Connectx-2 MHQH29C-XTR which is a dual port and I will put in the main ESXi box.

Now I'm looking at different scenarios but both have its pros and cons:
  1. Use the "main" ESXi box as a switch for 10GBE where all other SFP+ modules connect to. Run 10GBE Ethernet on it.
  2. Buy an infiniband switch then every device connects to the infiniband switch. Run infiniband on it.

This is my first experimentation for 10GBE/Infiniband. I need 10GBE for NFS/iSCSI for ESXi as well as running backups for ZFS file systems.

My main concern is: which cables should I buy? Should I get QSFP+ to 4xSFP+ breakout cables? Does Mellanox lock out their cables so you need to buy their brand or you can buy anyone you want?

Thank you very much in advance.
 
Last edited:
What are you using for your storage SAN/NAS?
Right now, FreeNAS doesn't have good Infiniband support. Doesn't even support the Mellanox cards.
 
Storage box is on ESXi.
NAS functions will be handled by a flavor of GNU/Linux (either Gentoo or Debian while SAN/NFS/iSCSI will be handled by OmniOS (OpenSolaris derivative).

Anyway if I do "point to point" (without switch) it would be 10GE ethernet, wouldn't it?
AFAIK the cards support both Ethernet and Infiniband, right?
 
I guess I'll do point to point connection using the configuration I illustrated above.

In that case I'll have like the following:
  1. ESXi Box 1 SFP+ -> Main Box (1 port of 4xSFP+ breakout cable to QSFP+)
  2. ESXi Box 2 SFP+ -> Main Box (1 port of 4xSFP+ breakout cable to QSFP+)
  3. Backup server SFP+ -> Main Box (1 port of 4xSFP+ breakout cable to QSFP+)
  4. Desktop SFP+ -> Main Box (1 port of 4xSFP+ breakout cable to QSFP+)

In this case can the main box have a single connection to every of the other boxes?

And, more importantly, could say ESXi Box1 access ESXi Box2 at 10GBE speeds? I don't think so since say Pfsense struggles to handle switching at that speeds ...


Basically I think this configuration is good for point-to-point but wouldn't be very good for anything else. Still not too bad considering one X540-T1 is 400€ and one 8-port switch like the XS708E is 800€-1000€.

So let's say for 4 aux boxes + Main with point to point I would need:
- 1 x Mellanox Connectx-2 MHQH29C-XTR for main box -> 100€
- 4 x Mellanox Connectx-2 MNPA19-XTR -> 100€
- 1 x QSFP+ to 4xSFP+ breakout Direct Attached Cable 5m -> About 100€
Total: 300€

With the switch:
- 1 x XS708E -> 800€
- 5 x X540-T1 -> 2000€
- 4 x Cat7 cables -> About 100€
Total: about 2900€

Fair enough one is point-to-point while the other is using switch.
Isn't there a way to do switching using Pfsense at intermediate speeds (say something like 3-4 Gbps)?

What would you reccomend?
Thanks for your help!
 
I originally went for the XS708E- and instantly regretted it (lack of ports, poor/lack of web interface/cli/featureset). Then stepped up to a dell 8024 (24port for about twice the price of the netgear) and find it a lot better!

However... that being said, wish i'd gone for fiber or infiniband.
 
I just saw that QSFP -> 4xSFP+ splitter is only supported on Mellanox switches, not on Mellanox NICs such as the ConnectX-2.

IPoIB (IP over InfiniBand) seems like a good option if integrating a switch ...

ltickett said:
However... that being said, wish i'd gone for fiber or infiniband.
Click to expand...

Why if I may ask?
 
Price probably reason #1
Also... ethernet seems a lot less popular, i'm sure that's not a coincidence?
Oh, and I believe ethernet also consumes more power...

Sure there's more reasons i can't think of
 
I just purchased a Voltaire InfiniBand 4036 Infiniband swich for what I consider a steal for that kind of hardware (400$ but with shipping&duties it became 480€).

As far sa I understand I now need some Mellanox Connectx-2 VPI cards for use with this switch, right? The Mellanox Connectx-2 MNPA19-XTR I own already are 10gbe ethernet cards, therefore not suitable with that switch that is Infiniband. Am I wrong?

Which VPI cards would you suggest? The Voltaire switch uses QSFP connectors. I would like to avoid "strange" cables like CX4<-->QSFP or QSFP<-->SFP+ breakout cables ...
 
Bump. Anybody has any ideas on which adapters to pair with this switch ?
I don't think the Mellanox I own are compatible ...
 
You must log in or register to reply here.
Back
Top