STP/LCAP to switch connections??

R

rsq

Limp Gawd
Joined
Jan 11, 2010
Messages
246
I need a sanity check on my idea.

I have a 10Gb/s Ethernet switch and a 1Gb/s Ethernet switch. The 10Gb/s switch is noisy, and hot. The 1Gb/s Ethernet switch is not.

All my machines (7) have both 10Gb/s card and 1Gb/s card.

I am looking for a way to put both switches in parallel, using STP or LACP or whatever to be able to transparently switch between the two switches. Basically, I want everything to run on the 1Gb/s switch until I need more bandwidth, at which point I power up the 10Gb/s switch. The fast switch should then take over transparently. I want only 1 single IP per machine, traffic should switch automatically, preferably without the TCP connections noticing that the fast switch took over.

I use only Ubuntu Linux as OS.

Is what I want to do possible? If yes, how should I go about setting this up? I am capable of researching the technologies on my own (I know LACP very well already) but I don't know how to start this one.
 
No. The switches would need to be stacked or use something like VSS. Even then you can't mix interface speeds in a LACP bundle.
 
You need to address two items here:

1. Server network redundancy. You will need to have each server handle changing between each switch if an interface goes down. Lucky you're on Linux and this is easy. Setup a NIC bond team with one 10gig and one 1gig NIC. Set the bonding mode to failover (mode 1 if i recall right) and set the 10gig NIC as the primary. That way your servers will all automatically switch between the two switches depending on if the link is up or down. You can't use LACP for what you want to do on the server side since i doubt your switches can stack.
2. Wiring the switches. This actually really simple, if the 1gig will always be up, have your 10gig connect to the 1gig switch with as many links as you deem important, and then have your 1gig switch connect to your router/firewall

Majority of what you want relies on the servers to make the determination here and cannot be done by the switches. Only way the switches could do that redundancy is if your switches were stacked or are nexus fabric and can create port-channels across physical switches due to a shared control plane
 
DermicSavage, I looked into the bonding mode 1 and it looks promising. Thanks for the tip.

Just out of curiosity, is there a way to do this in Windows, or is a real OS needed?

Will test it as soon as my data transfer completes. (5TiB zfs send/receive)
 
Windows natively supports NIC teaming in 2012R2 and later (tlb, lacp, and failover only for the OS level). It is pretty common for drivers in prior versions to provide the teaming functions but ymmv

If you do set up teaming in 2012R2, then when you create the team it will be set for 'switch independent' mode with an adaptive policy (or hyperv port if it's running hyperv) and set the 1gig adapter as the standby. Same effect as bonding mode 1
 
Last edited:
Noisy and hot, it's probably noisy because it's hot. What is your room cooling like? Where are the intake/exhaust on the switch and how do they relate to the room. Is the device on its own, in a rack, sandwiched etc - is there room to breathe. Any switch firmware upgrades?
 
BunnyKillBot said:
Noisy and hot, it's probably noisy because it's hot. What is your room cooling like? Where are the intake/exhaust on the switch and how do they relate to the room. Is the device on its own, in a rack, sandwiched etc - is there room to breathe. Any switch firmware upgrades?
Click to expand...

It's noisy and hot because it eats 200-300 watts of power. This is why a 32 port high speed switch (24x10Gig Ethernet + 8x8Gig fibre channel) is affordable on Ebay.

I love this thing, but not enough to keep it running 24/7.
 
I have tried the bonding mode 1 and it works, but not seamlessly.

When my switch boots, the link goes up before the switch actually forwards packets. All connections are dropped, and re-established after that.

I will look again at Spanning Tree Protocol to try and have a seamless transition.
 
Spanning Tree Protocol isn't going to get you what you want. STP is only for the failure of switches and/or with multiple links between them. With STP, you can have multiple links between switches. If a rat chews through one wire, the switch detects the dead cable and switches to a backup. There would be no error detection or a way to control which port you want to use at the Computer level unless you unplug the 1GB link. STP still has a place in today's world, but not as much as stacking or LACP over multiple links at the switch side.

If you want to use the 10gb for transferring large files and the 1gb for just regular internet, to me the easiest way it to have multiple IP addresses on each NIC and do a shared mapping using the specific IP address over the 10gb link. That way when you go to copy something into that share, it has to go over the 10gb link rather than the 1gb link.

EG, Give your 1gb link a 192.168.2 address and your 10gb a 192.168.5 address, then map the remote folder specifying the IP address. Depending on your switch, you could even go 1 step further to guarantee that the traffic would go over the 10gb link with a separate VLAN. I have 2 NICs in my computer all with separate addresses for what I need. General access and NFS mappings.
 
2 subnets is exactly what I don't want :) I had that before and its a hassle.

The bonding setup works fine. All I would need now is some rule to delay the switch over from secondary to primary. So far I have not found an option to configure this, but I keep on it :)

If needed I will try to script something, or send out some kernel feature requests...

It's nearly 2016, it's ok to expect a lot from technology. :)
 
You must log in or register to reply here.
Back
Top