Multi-site network routing advice

One of the schools that I work with has an entirely flat network structure (I have mentioned this before). We're looking into options for breaking up this large broadcast domain. The network consists of 3 school buildings and 3 administrative buildings. The high school is the "center" of the network, most of the servers are located there, and that is where the internet connection is. All the other buildings connect to the high school over private fiber, which is plugged directly into several switches.

I'm wondering if it would be best to put a small router at each site, or one larger router at the high school and terminate all the fiber into that one device? Does anyone have any experience or opinions on this?

The way I see it, having one central router would be cheaper, easier to manage, but introduce a single point of failure (if that router goes down, all sites lose network access). Having a router at each site would cost more and create more work, but it would give us an edge device at every site that we could access remotely for troubleshooting purposes.

I contract for a local school system of 17 schools. All of their traffic comes into a central building via point to point wireless. They currently have a single Cisco router that handles the traffic, and a switch configured to VLAN off each school. They are moving to using two Sonicwall devices in HA to provide the redundancy, and will likely move the VLAN config to those.

A lot would depend on the size of the schools, and budget.

The administration subnet should be separate from the "student" subnet by firewalls.
The last one I set up we had 1 admin and 2 student subnets per building.
Admin was wired and one student was wired the other wireless.
Teachers would connect to the wireless and vpn to the admin side.
The class room teacher ports used 802.1X with radius to help keep students from connecting a computer to the admin network.

You didn't say what brand router/switches you were running, but Cisco routers have HSRP for standby

Also use client isolation on wireless and protected ports on wired to help keep infections in check. Its also not a bad idea to enforce anti-virus on the end-point.

How are your buildings connected? Do you own the fiber links, or lease them from a company? That will change your design a lot. Are they 1GB, 10GB, or less?

Hardware:
Cisco 3750X (3560x if you are under a budget) as Layer 3 for each site

You said the majority of the servers are located at the high school, not all. To me that means you need dual redundant routers at the main location then a single router at the other buildings. If you loose a link that means some things will remain working.

Each building should have 6 subnets(separated by vlans), teacher wired, teacher wireless, student wired, student wireless, management and VoIP. Even if you aren't using all of this now, plan for it.

From there things will depend on how many devices you have connected at once. Personally there any many networks I deal with that need more than a 24-bit broadcast domain. Once you get over 254 devices per vlan organized addressing can get a little complicated but not terrible.

Ospf handles the routing table.

imho voip phones need to be on its own switch stack and subnet with a router.
The schools we did used pots phones but we did run extra runs for future voip phones as well as enough rack space to add the gear.

stormy1 said:

imho voip phones need to be on its own switch stack and subnet with a router.
The schools we did used pots phones but we did run extra runs for future voip phones as well as enough rack space to add the gear.

Click to expand...

I've found that as long as a good router is used I can get away with using vlans on a switch as long as a dedicated uplink for VoIP is provided. For example 1 router with 1 link to a switch for access and another physical link for VoIP in the MDF. From there I maintain 2 physical links (or a LAG with 2 links for access and a second physical link for VoIP) to the IDF's also.

Depends on the situation. Some closets just don't need more than 48 ports.

QOS or 2 separate links back to the main site is a must with VoIP, dedicated link is preferred if you are dealing with upwards of 50 phones or 100mbit links.

bds1904 said:

I've found that as long as a good router is used I can get away with using vlans on a switch as long as a dedicated uplink for VoIP is provided. For example 1 router with 1 link to a switch for access and another physical link for VoIP in the MDF. From there I maintain 2 physical links (or a LAG with 2 links for access and a second physical link for VoIP) to the IDF's also.

Depends on the situation. Some closets just don't need more than 48 ports.

QOS or 2 separate links back to the main site is a must with VoIP, dedicated link is preferred if you are dealing with upwards of 50 phones or 100mbit links.

Click to expand...

I have clients who are doing video training and or meetings locally.
The school I mentioned above is doing steaming video sessions to the desktop and or projectors from a local server plus interactive training and testing.
100 video streams on a switch stack can bring a switch stack to its knees even without voip.
Add voip and your headaches just got much bigger.
That kind of thing is going to get more and more common.

Another advantage of separate stacks is you can use lower end switches.

The worst however is plugging the pc into the phones......
Whoever came up with that idea needs to be drawn and quartered.

If it truly is secured private fiber, L3 switches like a 3750-x at each site and use OSPF between them.

wow, lots of posts. Thanks everyone!

boss6021 said:

I contract for a local school system of 17 schools. All of their traffic comes into a central building via point to point wireless. They currently have a single Cisco router that handles the traffic, and a switch configured to VLAN off each school. They are moving to using two Sonicwall devices in HA to provide the redundancy, and will likely move the VLAN config to those.

A lot would depend on the size of the schools, and budget.

Click to expand...

Its a small school district, ~2300 users, plus 100 guests, 300 guests during events.

stormy1 said:

The administration subnet should be separate from the "student" subnet by firewalls.
The last one I set up we had 1 admin and 2 student subnets per building.
Admin was wired and one student was wired the other wireless.
Teachers would connect to the wireless and vpn to the admin side.
The class room teacher ports used 802.1X with radius to help keep students from connecting a computer to the admin network.

Click to expand...

This is a terminal server environment, so there can only be one subnet. Any user can log into any terminal. Each school has several servers that provide an image for the terminal to boot from, while authentication and user profiles are handled by a server in the high school.

Wireless is split into two SSIDs. A "guest" SSID that blocks access to all local resources, and a "teacher" SSID with WPA2 that allows access to the LAN.

Radius authentication with 802.1X is a good idea, but our current switches don't support it (working on this). Right now all we have is a MAC address whitelist on the DHCP server, so all a student needs to do is statically assign an IP address and they're on the network. We're using a /8, so the chances of them picking a random IP that is unused are pretty good.

stormy1 said:

Also use client isolation on wireless and protected ports on wired to help keep infections in check. Its also not a bad idea to enforce anti-virus on the end-point.

Click to expand...

The end points are all Linux terminals. We do run anti-virus scans on the profile server and on the Windows terminal servers.

thrash408 said:

How are your buildings connected? Do you own the fiber links, or lease them from a company? That will change your design a lot. Are they 1GB, 10GB, or less?

Hardware:
Cisco 3750X (3560x if you are under a budget) as Layer 3 for each site

Click to expand...

The buildings are connected by our own fiber, direct point to point. We own the fiber and the land it was trenched through. We don't lease any strands or use VPLS or anything. Three buildings use multimode, three use single mode.

We're definitely under a budget.

bds1904 said:

You said the majority of the servers are located at the high school, not all. To me that means you need dual redundant routers at the main location then a single router at the other buildings. If you loose a link that means some things will remain working.

Each building should have 6 subnets(separated by vlans), teacher wired, teacher wireless, student wired, student wireless, management and VoIP. Even if you aren't using all of this now, plan for it.

From there things will depend on how many devices you have connected at once. Personally there any many networks I deal with that need more than a 24-bit broadcast domain. Once you get over 254 devices per vlan organized addressing can get a little complicated but not terrible.

Ospf handles the routing table.

Click to expand...

I have been planning to use OSPF if we go with a multiple router solution. Redundant routers at the main location is a good plan, if we can afford it. Do you recommend any technologies in particular for that?

stormy1 said:

imho voip phones need to be on its own switch stack and subnet with a router.
The schools we did used pots phones but we did run extra runs for future voip phones as well as enough rack space to add the gear.

Click to expand...

We're currently using an analog phone system (Avaya). We're in the planning stages of a VOIP rollout right now. This is one of the reasons why I want to get the broadcast domain under control now, rather than later. One of the other schools that I work for uses separate switches for voice and data, and it seems to work well. Their system is entirely outsourced though, so I can see how its set up.

bds1904 said:

I've found that as long as a good router is used I can get away with using vlans on a switch as long as a dedicated uplink for VoIP is provided. For example 1 router with 1 link to a switch for access and another physical link for VoIP in the MDF. From there I maintain 2 physical links (or a LAG with 2 links for access and a second physical link for VoIP) to the IDF's also.

Depends on the situation. Some closets just don't need more than 48 ports.

QOS or 2 separate links back to the main site is a must with VoIP, dedicated link is preferred if you are dealing with upwards of 50 phones or 100mbit links.

Click to expand...

So, data and VOIP on the same switch using VLANs, but provide a second uplink to the router for just VOIP? Thats also a good option if we can't afford the switching hardware.

stormy1 said:

I have clients who are doing video training and or meetings locally.
The school I mentioned above is doing steaming video sessions to the desktop and or projectors from a local server plus interactive training and testing.
100 video streams on a switch stack can bring a switch stack to its knees even without voip.
Add voip and your headaches just got much bigger.
That kind of thing is going to get more and more common.

Another advantage of separate stacks is you can use lower end switches.

The worst however is plugging the pc into the phones......
Whoever came up with that idea needs to be drawn and quartered.

Click to expand...

I'm not going to say we were planning on plugging the PCs into the phones......but we were planning on plugging the PCs into the phones.

Is this just an issue with bandwidth, or is there some other reason?

/usr/home said:

If it truly is secured private fiber, L3 switches like a 3750-x at each site and use OSPF between them.

Click to expand...

A 3750-x is approximately 1/3 of my yearly budget. We've pulled off so many miracles with FOSS in this district, the higher ups think we can manufacture hardware with our minds.

Jake said:

You didn't say what brand router/switches you were running, but Cisco routers have HSRP for standby

Click to expand...

I missed this post in my giant quote. I'm hoping to use Ubiquiti Edgerouters, since I use them with my WISP and am familiar with them. A ER-Pro should easily be able to handle the traffic this school district sees, and an ER-Lite would be enough for each location (most of the fiber is only 100mbps, a couple are 1gbps) with no firewall or NAT required. If I go with EdgeRouters i'll try to talk them into using an ER-8 at each school.

Based on you owning the fiber, I would just set a VLAN up for each building, trunk all of your fiber ports, and set your access ports at each site to the specific vlan you set for each building. What wireless do you have? Does it tunnel back to a controller, or do you do local switching at the ap level(flexconnect)? If it tunnels back, you can set interfaces up on the controller and use ap groups to give each building it's on vlan at the wireless level. This is all assuming you either have cisco wireless, or are willing to get it.


FYI, if you are running a district that size with a $30k budget, you need to have a serious talk with your super. I have a excel spreadsheet that i built, that allows me to plug in numbers of computers, switches, servers, printers, smartboards, wireless, and it auto calculates what a "decent" budget would be. If you want t his PM me and i'll send it over. I use it at every district i manage, and 80% of the time they give me that budget

wizdum said:

A 3750-x is approximately 1/3 of my yearly budget.

Click to expand...

Any recommendation I would give just went out the window. Knowing that, I simply would not have taken the job.

wizdum said:

A 3750-x is approximately 1/3 of my yearly budget. We've pulled off so many miracles with FOSS in this district, the higher ups think we can manufacture hardware with our minds.

Click to expand...

Then get a used L3 switch like a Nortel 5510 and buy a few spares or something like that if you are constrained on budget.

thrash408 said:

Based on you owning the fiber, I would just set a VLAN up for each building, trunk all of your fiber ports, and set your access ports at each site to the specific vlan you set for each building. What wireless do you have? Does it tunnel back to a controller, or do you do local switching at the ap level(flexconnect)? If it tunnels back, you can set interfaces up on the controller and use ap groups to give each building it's on vlan at the wireless level. This is all assuming you either have cisco wireless, or are willing to get it.


FYI, if you are running a district that size with a $30k budget, you need to have a serious talk with your super. I have a excel spreadsheet that i built, that allows me to plug in numbers of computers, switches, servers, printers, smartboards, wireless, and it auto calculates what a "decent" budget would be. If you want t his PM me and i'll send it over. I use it at every district i manage, and 80% of the time they give me that budget

Click to expand...

I'll do that, thanks. Even if they don't like the number, its good for them to see what the real world is like every once in a while.

Valnar said:

Any recommendation I would give just went out the window. Knowing that, I simply would not have taken the job.

Click to expand...

Yeah, I get that a lot. Money gets in the way a lot, but on the other side, as a tax payer it's nice to see the school trying to save money, instead of buying the newest shiny tech toy.

/usr/home said:

Then get a used L3 switch like a Nortel 5510 and buy a few spares or something like that if you are constrained on budget.

Click to expand...

I have heard good things about these switches, i'll add them to the list, thanks.

wizdum said:

I'm not going to say we were planning on plugging the PCs into the phones......but we were planning on plugging the PCs into the phones.

Is this just an issue with bandwidth, or is there some other reason?

Click to expand...

Its not bandwidth there is enough bandwidth at 1gb it is time slots.
Voip requires a continues stream of bits to work.
If the line is busy transferring other stuff at that same moment even though you have a lot of bandwidth the phone can stutter.
This is particularly bad when watching streaming video or listening to streaming music but a large download can cause issues.
One place that used to do it could always tell it was the day after patch Tuesday.
The voip was local so it was not the internet connection.

Also terminal services can act like a streaming connection in bursts.
I would not bet against it causing issues.

Some phones are 100mbit which is even worse because it has fewer time slots.
Overall its a very bad idea.

stormy1 said:

Its not bandwidth there is enough bandwidth at 1gb it is time slots.
Voip requires a continues stream of bits to work.
If the line is busy transferring other stuff at that same moment even though you have a lot of bandwidth the phone can stutter.
This is particularly bad when watching streaming video or listening to streaming music but a large download can cause issues.
One place that used to do it could always tell it was the day after patch Tuesday.
The voip was local so it was not the internet connection.

Also terminal services can act like a streaming connection in bursts.
I would not bet against it causing issues.

Some phones are 100mbit which is even worse because it has fewer time slots.
Overall its a very bad idea.

Click to expand...

Yep. Looks like i'm going to have to do some cable drops for that project.

wizdum said:

This is a terminal server environment, so there can only be one subnet. Any user can log into any terminal. Each school has several servers that provide an image for the terminal to boot from, while authentication and user profiles are handled by a server in the high school.

Click to expand...

That is not true TS has no problem crossing a properly configured router and firewall.
Allowing only TS connections between subnets is a good practice in a TS environment.
I have put all the TS servers on their own subnet and used firewalls to block all access other than TS a few times.
In one that was for a testing lab the TS had access to windows update and the authentication server needed ports and the PCs had access to nothing beyond the local network and were set to auto login and store nothing between logins.
We used software on the TS to push windows and AV updates the the PC's.

stormy1 said:

That is not true TS has no problem crossing a properly configured router and firewall.
Allowing only TS connections between subnets is a good practice in a TS environment.
I have put all the TS servers on their own subnet and used firewalls to block all access other than TS a few times.
In one that was for a testing lab the TS had access to windows update and the authentication server needed ports and the PCs had access to nothing beyond the local network and were set to auto login and store nothing between logins.
We used software on the TS to push windows and AV updates the the PC's.

Click to expand...

I didn't fully explain that properly. The issue isn't traversing multiple subnets, its assigning users to different subnets. We're using NIS (*pause while I wait for the laughter to subside*). Its very basic, and doesn't support the ability to place students and teachers on separate subnets.

I hope to replace this with some sort of modern directory service this summer. Step one was completed last week, which was to update the NIS server from RHEL4 to Ubuntu 14.04.

What does the budget look like for this solution?

wizdum said:

I didn't fully explain that properly. The issue isn't traversing multiple subnets, its assigning users to different subnets. We're using NIS (*pause while I wait for the laughter to subside*). Its very basic, and doesn't support the ability to place students and teachers on separate subnets.

I hope to replace this with some sort of modern directory service this summer. Step one was completed last week, which was to update the NIS server from RHEL4 to Ubuntu 14.04.

Click to expand...

2 ways to do it.
repeaters on subnets or named nis server list on the client.

dmolter said:

What does the budget look like for this solution?

Click to expand...

For the changeover from NIS to something newer? $0. The hardware has already been upgraded, we just need setup the software.

stormy1 said:

2 ways to do it.
repeaters on subnets or named nis server list on the client.

Click to expand...

I'm still not following. The image is downloaded to the terminal before the user logs in. The only thing that changes between a student log in and a teacher/administrator log in, are the files and settings in the users /home directory.

I don't really know what NIS is... but it seems really odd to me that you can't route with it...

Why do students or teachers need to be assigned to subnets?

goodcooper said:

I don't really know what NIS is... but it seems really odd to me that you can't route with it...

Why do students or teachers need to be assigned to subnets?

Click to expand...

It seems to be a general "best practice". I don't think it really applies to my situation since there are no shared resources that teachers/admins have access to that students do not. I can see separating the cameras, phones, and management devices with VLANs though.

NIS is a very basic directory for users and groups. It basically just shares the /etc/passwd (user names and passwords) and /etc/groups (groups) files from a "master" server to other clients. /home is shared out via NFS, so that once you authenticate with NIS, you can access your files. NIS traverses routed networks fine, but there is no real user profile like you have with a modern directory.

wizdum said:

It seems to be a general "best practice". I don't think it really applies to my situation since there are no shared resources that teachers/admins have access to that students do not. I can see separating the cameras, phones, and management devices with VLANs though.

NIS is a very basic directory for users and groups. It basically just shares the /etc/passwd (user names and passwords) and /etc/groups (groups) files from a "master" server to other clients. /home is shared out via NFS, so that once you authenticate with NIS, you can access your files. NIS traverses routed networks fine, but there is no real user profile like you have with a modern directory.

Click to expand...

Yea I think your best practices are pretty much out the window if your entire network is on a single huge subnet