Redesigning switch layout for redundancy.

U

USF-Nealio

Limp Gawd
Joined
Jan 18, 2012
Messages
373
My current switch layout at work is very basic and offers no redundancy. I'd like to at least setup redundant core switches, so that if any of my switches would to fail, it would result in a smaller number of systems affected.

Current Core Switch: HP Procurve 3500yl-48G (3500yl-24G as cold spare)
Internet Switch: 3com (forget model)
User distribution switches: 9 Planet 10/100 Layer 2 Managed switches, and 2 Dell Powerconnect 5448
Server switches: 5 Procurve 2810-24Gs (1 in each server rack)

In addition, we have 50+ site to site VPNs using different 192.168.x.x subnets across 2 different VPN concentrators, several routers, and a pair of MPLS's.

My first step to help improve speed and reliance was to reconnect the user switches. The old distribution switches were daisy chained, and all users 120 users were on a single 1GB port on the core switch, and if switch 2 of 11 failed, I'd have 10 switches knocked out of the network. I've since connected them directly to the core switch, and the users have noticed the improvement.

I'll try to get a diagram, but it's basically a large hub and spoke. Each and every switch is connected directly to the single core switch. All of the internet sources are connected directly to the 3com and are on separate VLANs which are then connected to the core via a trunk, which then handles the routing. Currently, I'm using less than half of the ports on the current core switch, so I should still have enough ports on the 24G to at least maintain the critical systems. Each server switch is trunked to the core, as is the internet switch, but the user network is not (but probably should be). Instead they are connected via simple untagged uplinks.

I absolutely cannot afford new equipment right now, so whatever I do will require reusing existing equipment. I'd very much like to increase the bandwidth from the server switches to the core if at all possible. Is there an easy way to configure and Active-Active setup that would improve bandwidth? Or would It be active-passive and require a 2 port LAG on the 48 port (for speed), and a 3rd port for the 24 port (for redundancy).

NOTE: With any luck, we'll be dropping our analog phone system and replacing it with VOIP sometime within the next year. So I should be getting some extra budget to replace the distribution switches with POE. Until then, I'll have to live with my crappy Planets.
 
distributi.jpg


Something like this. Your switches all need to support spanning tree though.
I would put the servers on the distribution switches.

You could even have multiple links between switches for more redundancy there as well. Something like etherchannel would work well as well.
 
/usr/home said:
Something like this. Your switches all need to support spanning tree though.
I would put the servers on the distribution switches.

You could even have multiple links between switches for more redundancy there as well. Something like etherchannel would work well as well.
Click to expand...

Thanks for the input, but why wouldn't you reuse the switches I have in the server racks? I have 40 physical servers (not counting other equipment) across the 5 racks, so I don't think they'll all be able to fit onto the 2 distribution switches. Each servers generally has 3 nics (1 iLo, 1 SAN, and 1 LAN) all being connected to the 2810s I have in each rack. Obviously each network is VLANed off from one another, and each switch is then trunked to and from the core switch.
 
Then I would add all 5 of the "server" switches to the "distribution layer". Not sure if you would have the port count to do it though.

Someone with more experience may be able to chime in as well.
 
We are currently using a topology similar to the picture above. However our distribution are in a VSS pair for each (making it 4 total distribution switches for redundancy) . We are currently in the works of hanging a nexus switch onto our 6500s to house our server farms.
 
There are various factors for why one would choose one setup over another.

In this case I would (if possible) set it up something like:

*) Use the drawing /usr/home made as a base.

*) What /usr/home calls distribution layer I would in this case call core-layer so you use 3500yl-48G + 3500yl-24G as Core1 and Core2.

So you will have access-layer connected directly to core-layer without a distribution-layer in between (simply because in this case there is no need for additional unnecessary hardware).

*) Connect Core1 and Core2 to each other with as many interfaces as possible, 4 or even 8 if you can afford it. Set this up as a static etherchannel/lacp - dont forget to verify if you can change the loadbalance algoritm from srcmac+dstmac into srcip+srcport+dstip+dstport (to better utilize the interfaces in the channelgroup).

*) Connect each server/client switch (2810-24Gs) with dual channelgroups. Like two interfaces to Core1 which forms trunk-01 and two interfaces to Core2 who forms trunk-02 (and same thing here, change loadbalancing algo if possible). If we look at /usr/home drawing each line will basically be doubled.

*) Enable rapid spanning-tree along with portfast where needed. Also make sure to not allow spanningtree from servers/clients (unless needed).

*) Verify that your redundancy works as expected by setting up a testprotocol which you will follow and document the result (including how long time it takes for the network to convergate).

Reason for using rapid spanning-tree instead of old one is that a broken link will take roughly 2 seconds to fix instead of roughly 30 seconds. Also by using channelgroups (trunk-01 and trunk-02 and the trunks between Core1 and Core2) will make your network more robust.

However you should at the same time also include some sort of monitoring (SNMPv2c or SNMPv3) because these extra cables wont help you if first one interface breaks and then it goes a couple of months before next one breaks until all uplinks are down. Using channelgroups (and spanningtree) will make the network stay up in case of bad things happens but at the same time you need to fix the bad things (like bad cable or bad interface) otherwise your redundancy is broken yet again.

There are various SIEM's to use as monitoring on the market and its a matter of taste which you prefer but I could recommend you to test OSSIM which is available in a free community edition at http://communities.alienvault.com/community/

As mentioned a more modern setup would be if Core1 and Core2 can be setup as virtual chassis (VSS, different manufacturers have their own name for this). This way Core1 and Core2 will be two physical units but act as a single one when it comes to channelgroups. This means that the accesslayer (2810-24Gs) only need to setup a single trunk with (for example) two cables towards Core1 and two cables towards Core2 and use all four cables at once (otherwise with spanning-tree only one pair will be used - the other will be in "shutdown" mode until the other pair fails).
 
You must log in or register to reply here.
Back
Top