[H] network engineers, Please help me solve an issue

K

KaosDG

[H]F Junkie
Joined
Apr 3, 2000
Messages
11,939
We're having a problem with bandwidth utilization / VPN efficiency.

Disclaimer: I've had NOTHING to do with this initial setup, I'm just taking over this shitstorm. also, i'm not well versed in WAN setups so please tell me if i'm being an idiot.
That and the fancy visio stuff is to show the non-technical owners.

Our network currently serves 2 companies, Company A (the main company) and Company B (sister / partner company) and they share our internet pipe. (2 bonded T1's)

We are on the same subnet (historically, the second company was a subdivision but split off in the past 2 years)

Now, this is not really an issue except for the following:
Company B has a remote office in Stamford, CT.
We have a VPN to that office, over our internet pipe.
Initially, ALL of company B's servers were in our NY office, but due to "slowness" it was decided (much to my dismay) to move them to CT. (Only relocated the problem, didn't solve it)
Now all of B's servers are in CT.


Right now, B's NY Office is experiencing the major slowdowns. To alleviate it somewhat, we setup a second Exchange server in NY for the NY employees, and that saves them from the headache that is Outlook.
However, their main file server is still in CT.
That means all their home directories, working files, etc have to be pulled across the VPN.

This makes me sad.
Why?

Because Company A has some remote users (plain ol' windows VPN connection) that have to access their work from home.

During the day, during Peak work hours, the remote users are constantly losing connection, company B is constantly complaining of slow file access, and so on.

We will further eliminate some bandwidth usage by setting up a seperate Fileserver for B, in the NY office. We can segregate their files and not worry about replication, because they don't usually work on the same items. (We'll do local backups as well)
If they do, it's a small amount of data (usually a word / excel doc or 3 for administration) and they can connect directly to either fileserver to get it.

Company B also has an MS-SQL based application that they need to access (that server is in CT as well)



Here's where I need your help.

I have 3 different scenarios to present, and would like to know what your best recommendations are.


My first thought was to completely segregate A and B;
Have company B get their own Internet T1 in NY, and then have them get a point-to-point T1 for their connectivity between sites.



This way, NY will have it's own internet bandwidth, CT will have it's own internet bandwidth, and they have dedicated Bandwidth for files / e-mail and database that they all access.

This is costly, however. (Though either or both sites could go SDSL for internet, though at least one site needs a static IP)


My other option would be to again, segregate the 2 companies
have B's NY office get a T1, and run a VPN tunnel to CT again.

Of course, they will be putting internet bandwidth on that as well, but without company A sitting on the pipe, it should be much better.

This is less costly than the first proposal.



My last option would be to get a seperate SDSL line installed at the NY office, strictly for a remote user VPN.

This will quiet Company A's remote users, but as you see the other issues still remain.



My only saving grace in this is that Company B will be relocating sometime this summer, so we only have to deal with this for so often.

However, I am still charged with the task of getting company B setup efficiently and effectively.

Honestly, I think I should go with the Point-to-point T1 / individual internet pipes scenario.

Your thoughts?
 
Many different ways of tackling this...I'll relate to one of my bigger clients, a home healthcare agency that has 5x locations in CT.

I built their wide area network, using DSL in 4x locations, and cable in the last one I recently put in. (and WOW was that last location nice...at an awesome assisted living lodge for the "elders" of one of the tribes that has a mega casino)

Now a lot of this design (as are most of my networks) is based on the functionality of the application they run. The particular scheduling/patient tracking software my client uses...is designed for multi-site use...and has "data exchange" functionality. So their big central office...I have them on a 6,000/384 DSL pipe. I have their main server there, running SQL and Exchange, antivirus head server too. Sonicwall router. I'll call this office "mothership".

The 4x satellite offices...are on 1500/128 connections...Sonicwall routers also. Each satellite office uses the Sonicwall router to establish a "router to router" VPN tunnel to "mothership".

So basically look at the wide area network as a "hub and spoke" design.

Their software....each satellite office has what I'll call a "mini-server". This mini-server runs their scheduling software on MSDE (it used to all be Visual Fox Pro...they upgraded engines a couple of years ago, but even back on VFoxPro the functionality was the same). This way the clients at the satellite office can run from their "mini-server". Every 15 minutes...the main server at mothership drops a "seed" to each satellite mini-server...the mini-server feeds the seed with updates..then 3 minutes later the main server at mothership picks up the package...and imports it into the main database at mothership.

The satellite offices also can run Outlook tied into the Exchange server just fine through the VPN tunnel.

The Sonicwall routers allow you to dedicate a percentage (high) of the broadband bandwidth to the VPN tunnel...so that way the secretaries don't kill the bandwidth running online radio or something.

Remote home users, and support like me, connect using the Sonicwall Gloval VPN client. One can connect to any office depending on what you need to get to. Instead of connecting to the main office...to shoot out through another VPN tunnel to a satellite...you just connect to the satellite.

To help tunnel speed, you're not allowing netbios traffic are you? Just use DNS.

Of course a LOT of this depends on who needs to get access to what, and what kind of flexibility you have in your software. Perhaps terminal services? Instead of trying to run SQL through a tunnel. Exchange server? Everyone one Outlook 2003 running in cached mode? Beats older Outlook as far as being able to run through a tiny pipe.
 
My first thoughts were why not just get a larger pipe. While looking at your diagram I started to think about our companies network. The company is huge and has offices all over the world. Every single site has multiple domain controllers, exchange, and files servers on site. We're talking about a much smaller company but if getting a larger pipe isnt cost effective, how expensive would it be to have your company provide the majority of networking services on site at each location? Did you get any quotes for more bandwith? Dont be mistaken above are just my initial thoughts on your question, I think the under lying question is how much does your company really want to invest in correcting the problem. Then you just need to see where placing that money would give you the most bang for your money.
 
So all of Company A's servers are in NY, but all of Company B's servers are in CT (except exchange?)?

I mnight look at using Terminal services or Citrix. it depends on how many users you have at each site?
 
I'd say consider terminal server and citrix.

You can also put devices capable of doing QoS at the remote offices (maybe an ASA5510 for bigger offices, or something like an 800-series Cisco router for the smaller offices) and limit the bandwidth they get for the Internet to 15-20k, and leave the rest for network access. It is very easy to setup policies like that, and they work extremely well. We did this where I used to work just before deploying Citrix and it works like a champ. The plus side is IF they get past our proxy filtering for internet radio or streaming video or whatnot, they still only get 15k at the remote sites to surf with, and that isn't much...not good enough for good video, and barely enough for low quality radio.

I'd work on managing/shaping your bandwidth before you look at putting in fatter pipes. You might stick MRTG out there to monitor your links and get a readout on how much you're using.
 
Thanks guys, I'm going to try to address some things, bear with me if I miss something (only had 2 cups of coffee yet)

Stonecat:

We do allow netbios, simply because of the way they are working now.
ALL of their files are stored in CT, so all the NY users need to map to a drive on the CT server. Their Home directories, public shares, etc are up there.

We could go with a terminal server for the SQL app, I hadn't really given that much consideration.

We only run Office 2000, so no cached exchange mode. (We're also running WinNT 4 servers also, so I'll let you all laugh at me now)


winuxgeek:
We thought about getting a larger pipe, but cost is a major issue, and it doesn't really solve the problem of how this system was setup in the first place.
And you hit the nail on the head with "think the under lying question is how much does your company really want to invest in correcting the problem."
They (like most privately owned smalkl business) want to solve the problem with spending NO money. (Unrealistic, which is why I have to spend so much time mulling options and details)


oakfan:
We have roughly 12 users in company B in CT, and about 20 in NY.


Boscoh:
I really wish we had QoS capabilities and shaped traffic.
But all we have in the NY office is an old watchguard FW, and linksys unmanaged switches. No SNMP on the firewall, so we can't readily see what's being used, and no manageability on the desktop ends.

We do have 2 ciscos (2924 & 3524 that are MINE) that I put in for some manageability. but good lord, this place is a circus.
 
From the information you have given I think Citrix would be perfect. However most small business aren't willing to fork out the money to set it up. for 20 users it would run you 17-18K for Hardware and Software.

I think the best soultion given the information you provided would be to move all the servers back to NY. Then setup Terminal Services. All users in CT would use the Terminal server over a P2P T1. Then the external users would VPN into a T1 (to the internet) in NY then use TS as well. This does a lot of good things. The RDP protocol doesn't use much bandwidth. All of your data is centeralized and never leaves the local network in NY.
 
KaosDG said:
We do allow netbios, simply because of the way they are working now.
ALL of their files are stored in CT, so all the NY users need to map to a drive on the CT server. Their Home directories, public shares, etc are up there.
Click to expand...

Not good for tunnel performance....netbios through a VPN tunnel really "clogs the pipes".

Forgive me if you mentioned it above..... but...

1) How many workstations at each site?
2) Is there a DC running at each site (ultimately..DNS)?
3) Any Win9X workstations..if so, servers at each site running WINS?
 
YeOldeStonecat said:
Not good for tunnel performance....netbios through a VPN tunnel really "clogs the pipes".

Forgive me if you mentioned it above..... but...

1) How many workstations at each site?
2) Is there a DC running at each site (ultimately..DNS)?
3) Any Win9X workstations..if so, servers at each site running WINS?
Click to expand...

Yeah I'm seeing the netbios backlash a lot.

1) about 35 workstations in NY, and about 20 in CT. (some of them aren't active, they have a training and testing area in each site)

2) Both the PDC and BDC are in CT. (Yeah, that's a real good decision)

3) No Win9x Workstations, only 2000/Xp. Though we runs WINS since it's NT4 backend.
 
I like all the suggestions above particularly the terminal services / Citrix , Newer client version of Outlook, and QOS .

All that being said you are guessing at what is really your problem. You need to get a better idea of what is taking up the pipe. "Boscoh" eluded to it by suggestion the use of MRTG. I'm no expert in MRTG, but I don’t think that would give you the granularity you would need . For example a conversation list with the bandwidth of each.(Please correct if I am wrong)

Before I gave any suggestion I would sniff the line (i.e. ethereal and only capture the first 64 Bytes), or a simple tool like Wallwatcher , or if you have Cisco Equipment RMON is also very useful

You can implement any of the solutions you have above, but without understanding what is really happening you will look the fool if it doesn’t work and have no idea why.
 
Oh...NT 4 only for servers? :(

What are the workstations using for DNS at the NY locations?

I'd look at any means of just using DNS/WINS/..even lmhosts file edits, for name resolutions, and kill that netbios traffic.
 
The biggest difference that I see in your diagrams is that your current setup (the first diagram) has sharp 90 degree bends in the traffic paths. All of your proposed solutions afterwards show nice, rounded arcs for any turn that the data path takes. This might explain your problems. The 0's can get around the sharp turns (they're round), but all your 1's are probabbly piling up on the wire. :p

In all seriousness, though, I think that it's a bad thing(tm) to have your PDC and BDC's in CT. Also, if there is any way that you could disable netbios across those WAN links, you will see big gains.

I'm not sure what tools are available to you for monitoring / modifying your users internet behavior, but in my experience, WHATEVER the size of the pipe, dipsh*ts always manage to find a way to piss all your bandwidth away. Since your company doesn't want to spend any money on the problem, you might consider taking something like an old workstation or some such and building up a proxy and / or traffic shaper to put between the offices. If you can cache the most used web content (and lock out streaming garbage and content seen as nonproductive by your employer... like, say, consumptionjunction) you will begin to regain control of what your bandwidth is being used for.

In my (enterprise, decent budget) experience, allowing USERS to share your internet pipe that is used for SERVING traffic (vpn users, external web applications, etc.) is a losing battle. The bandwidth needs for both will continue to grow and step on each other's toes and from the sound of it, you don't have any easy way to divide these needs.

If at all possible, it would be nice to set up a proxy in each office, hooked up to the cheapest DSL provider you can find and MAKE everyone in the company use that to get to teh intarweb. This will guarantee that your (expensive, low latency) T1s are used for REAL, BUSINESS CRITICAL uses rather than reading up on what scandal $CELEBRITYOFTHEWEEK is involved in.

Just my bitter opinion. Best of luck to you.
-q
 
YeOldeStonecat said:
Oh...NT 4 only for servers? :(

What are the workstations using for DNS at the NY locations?

I'd look at any means of just using DNS/WINS/..even lmhosts file edits, for name resolutions, and kill that netbios traffic.
Click to expand...


we have a local 2000 DNS server here, and WINS forwarding.
(Company A's WINS servers are used for Company B, as we still have the domain trust intact)
 
I have a similar problem like yours. From my experience, http traffic takes up a huge chunk of internet traffic. You can setup ntop and SPAN the ports to monitor which protocol is choking up your pipe. Is it your site to site VPN links or your http traffic? Seeing that you dont have managed switch, just hooked up the outside/untrusted interface of the firewall to a hub. www.ntop.org

Http traffic can make up to 90% of your internet traffic. Once you know what protocol is choking up your bandwith, you can come up with a better design and you can maybe implement a QOS.

And with these statistics generated from network traffic probe like ntop, you can better convince your management to get a bigger pipe, get a bigger budget etc.
 
KaosDG said:
we have a local 2000 DNS server here, and WINS forwarding.
(Company A's WINS servers are used for Company B, as we still have the domain trust intact)
Click to expand...

OK...so there's your answer to get rid of netbios passthrough on the VPN. Make sure all machines across the WAN are registered in DNS...and workstations are using that DNS as their primary. Then kill netbios passthrough on the VPN tunnels.

And just to double check things....seeing as there are some legacy machines around here...nobody at all is running NetBEUI or IPX, correct? Including print servers? IPX is chatty.
 
DaturaX said:
I have a similar problem like yours. From my experience, http traffic takes up a huge chunk of internet traffic. You can setup ntop and SPAN the ports to monitor which protocol is choking up your pipe. Is it your site to site VPN links or your http traffic? Seeing that you dont have managed switch, just hooked up the outside/untrusted interface of the firewall to a hub. www.ntop.org

Http traffic can make up to 90% of your internet traffic. Once you know what protocol is choking up your bandwith, you can come up with a better design and you can maybe implement a QOS.

And with these statistics generated from network traffic probe like ntop, you can better convince your management to get a bigger pipe, get a bigger budget etc.
Click to expand...


Hoo billy.

Just got ntop installed and running.


You were right sir, http traffic is right now 80.7% of all TCP/UDP (Only running for an hour)
 
The next thing to do is talk to management about a puiblished policy and monitoring for abuses. Basicaly you show management the %80, have them send out a dont use the internet for anything but work, and then you monitor and report on abusers. While QOS will help greatly, if anyone has any legitimate work to do on the web they will get the same crappy performance, unless it's clearly defined so it can be added to the QOS policy.
 
moetop said:
The next thing to do is talk to management about a puiblished policy and monitoring for abuses. Basicaly you show management the %80, have them send out a dont use the internet for anything but work, and then you monitor and report on abusers. While QOS will help greatly, if anyone has any legitimate work to do on the web they will get the same crappy performance, unless it's clearly defined so it can be added to the QOS policy.
Click to expand...

Thats very true.

What might be a good thing to do is to get a 30-day trial of Websense, install it in Monitor Only mode, and generate reports on what are the most commonly visited websites. It's quite possible that they're shopping websites, online photo galleries, or something which is tunneling streaming data over HTTP. If the problems continue after you get management involved and nothing seems to be working, your best option might be to put in a filtering system to block whatever non-work categories are taking up the biggest chunk of HTTP. You could also just limit them to 30 minutes or an hour a day, or open it up during lunch hours only.

If you can get people to surf mainly work-related sites, enforcing a QoS policy on HTTP traffic is going to be much more effective overall and have the least impact on business, because you'll be dealing with mainly business-oriented traffic and I would guess that the majority of your current HTTP traffic is not business related.
 
KaosDG said:
Hoo billy.

Just got ntop installed and running.


You were right sir, http traffic is right now 80.7% of all TCP/UDP (Only running for an hour)
Click to expand...

Yeah. I know how annoying it is for someone to be choking up your internet link by just 1 huge file download and slows everything else to a crawl. So now you know http traffic is the culprit.

This is why network monitoring tools are so important. If you dont have statistics to back your claims up, usually in the case for network administrator like us. It is very hard and it will be a guessing game.
 
So it looks like I might be installing a DSL line or something strictly for web browsing / non-business internet stuff here.

lord. they don't see the overall problem.


Squid here i come :(
 
I absolutely love it when non technical people decide whats better technically for the company despite how many good points you make on how bad their plan is. :(
 
jonw757 said:
I absolutely love it when non technical people decide whats better technically for the company despite how many good points you make on how bad their plan is. :(
Click to expand...

Best part is, a PROOF READER is the one demanding the new line, striclty for HER OWN USE

and they are giving it to her.


perhaps I should suggest a DS3 so everyone can share the wealth.


and people wonder why IT people are so ornery.
 
Depending on what she is doing on the 'net...you could potentially use the argument that her use of the Internet could land a massive lawsuit against the company and pose serious, serious legal problems for the owners. Just a thought...

That will be valid regardless of whether or not you use another connection, unless she is paying for it as an individual. Ignorant management types tend to perk their ears when you mention legal issues. They understand that legal means money.
 
Boscoh said:
Depending on what she is doing on the 'net...you could potentially use the argument that her use of the Internet could land a massive lawsuit against the company and pose serious, serious legal problems for the owners. Just a thought...

That will be valid regardless of whether or not you use another connection, unless she is paying for it as an individual. Ignorant management types tend to perk their ears when you mention legal issues. They understand that legal means money.
Click to expand...


Well they want the line for when she's at home, to vpn over.

Instead, I am going to setup that line for everyone's http/non-business traffic, therefore freeing up our T1's for the real work. (and saving money off the top with an ADSL connection since we don't need the synchronicity or SLA of a T1 or SDSL)

We're working with IPCop and a few addons (adv. proxy and urlfilter) right now to curtail the http abuse once we get the line in place.
 
KaosDG said:
So it looks like I might be installing a DSL line or something strictly for web browsing / non-business internet stuff here.

lord. they don't see the overall problem.


Squid here i come :(
Click to expand...

Good choice. I love squid. Now you can monitor your http access for your users and you will be the Big Daddy in your company. Now you can submit a weekly report on the sites that your users go to and you can post a Top 10 most visited Site for your company. :p
 
You must log in or register to reply here.
Back
Top