Linux software switching?

[dgingeri]

OK, so there's this feature in Windows Server 2008 and up that allows a system to be configured so that multiple network connections can be grouped into what is effectively a software switch. An outdated system and some older 10Gb NICs can effectively be turned into a poor man's 10Gb switch. It's great. However, Windows Server costs money, more money than is really useful for a real poor man's 10Gb switch.

I have managed to find some old Intel 10Gb CX4 NICs and short cables for fairly cheap. Cheaper than most 10Gb NICs, anyway. I had been using them as direct connects between some systems for iSCSI and a home version of a SAN, for learning a lot of the datacenter skills I need to advance my career. I recently set up two of these cards in one system along with a couple 1Gb NICs to make a software 10Gb switch to allow two systems to access the one storage server. I have the free version of VMWare's ESXi 5 Hypervisor, so I made a VM dedicated to this software switch while utilizing the remaining resources for other VMs. Right now, I'm running it with a trial version of Windows 2008 r2 server, but that has to be rebuilt every 60 days because the trial period runs out.

I was wondering if there is a way to use Linux, whatever version, to make a software switch like this. Is there a Linux app that can do this?

 

[plupien79]

Any Linux/unix router distro can do this out of the box.

It's dirty but it'd work. just don't think you'll get the full 10gb out of it.

 

[toast0]

This is pretty straight forward on Linux. You have at least 3 options:

1) Straightforward IP routing, configure Linux so you can ping all your hosts, configure all the hosts so with the Linux box as the default route, enable IP forwarding on Linux (sysctl net.ipv4.ip_forward=1; maybe some firewall rules), Tada.

2) Proxy arp: need to enable IP forwarding and firewall rules like step 1, but also enable proxy arp (sysctl net.ipv4.conf.all.proxy_arp = 1), and you might not need to change the host configuration.

3) Ethernet bridging: brctl addbr br0; brctl addif br0 eth0 eth1 etc

The only tricky thing is that since this has been doable for a long time on Linux, there are some obsolete howtos out there. Make sure you go from a howto with the current firewall technology (nftables is bleeding edge, iptables is probably OK too, ipchains/ipfwadmin are super old)

 

[dgingeri]

Well, I've done routing with iptables and ip6tables. That's definitely no fun. I don't understand what proxy arp is. I'll go do some research on that. That bridging looks promising, but I saw a couple pages that call it bridging, but in fact they were routing between subnets. So that kind of depends on the author's definition of bridging.

I don't need full 10Gbe, but at least getting an effective 5Gb would be nice.

I'll continue research on google searches. If anyone has anything else to suggest, I'm all ears.

 

[devman]

You shouldn't have to touch iptables if bridging is setup properly. Bridged packets should not enter the IP layer (they are not being routed), and this should be the best performing of the three options as it logically groups the bridged interfaces the same way a layer 2 switch works.

On some distro's this may require setting parameters in /etc/sysctl.conf

net.bridge.bridge-nf-call-ip6tables = 0 #Disable iptables processing of bridged IPv6 traffic.
net.bridge.bridge-nf-call-iptables = 0 #Disable iptables processing of IPv4 bridge traffic
net.bridge.bridge-nf-call-arptables = 0 #Disable arptable processing of bridged arp traffic

The basics are

1) Setup kernel parameters for bridging (mentioned above)
2) Create bridge with brctl (bridge-utils package on most distros). "brctl addbr br0"
3) Add interfaces to bridge. "brctl addif br0 eth0", "brctl addif br0 eth1"

Done. These steps will have to be incorporated in to how your distro of choice manages network interfaces (systemd, NetworkManager, Fedora network-scripts, etc), but you can run them on a root command line to test it out before configuring the bridge in to the system start up.