Buffer Bloat, T1 lines, and Cisco Vs. Linksys routers

T

tdowning

Gawd
Joined
Oct 7, 2000
Messages
518
I'm volunteering at a small non-profit.

They have a T1 internet connection, and I think it is suffering from massive buffer bloat.

The T1 endpoint (not sure of the terminology Re: T1 equipment) feeds an ethernet cable that goes into the WAN port of a Linksys BEFSX41 (HW version 2.1)

Code: 
ping www.google.com -n 100
(snip)
Ping statistics for 74.125.225.82:
    Packets: Sent = 100, Received = 97, Lost = 3 (3% loss),
Approximate round trip times in milli-seconds:
    Minimum = 14ms, Maximum = 537ms, Average = 172ms

several times, when the ping times were reasonable for an extended period of time, (less than about 50ms) I would start a speedtest.net test of internet speed. As soon as it started ping times would spike into the 400-600ms range, as well as get timed out pings. Additionally, the speedtest.net graph of upload speed would show a massive spike to 5Mb/s or so before sharply declining and coming out to an average of about 1.2Mb/s

This is the first time I have experienced a problem like this myself, but it sounds like buffer bloat. Linksys has apparently never published an updated firmware for these devices, so I don't have that option.

What I would like to know is:

If you deal with a T1 line:
1. Do ping times vary widely when saturating upload link, vs saturating download link, vs nominal to minimal usage
2. What are you using for a NAT/Firewall/Router
3. Do you think that replacing the Linksys with a Cisco ASA model would fix this issue
4. Do you know if Cisco devices support CoDel (Controlled Delay the protocol designed to minimize buffer bloat issues?

Thanks for your help.
 
Last edited:
I deal with 4 bonded T1's and 2 other standard T1's.

To answer some of your Q's:

1. No our pings are very steady even under load. What router are you running and have you had your ISP look at it and run some tests against it? I have run into plenty of strange issues with leased lines and the ISP is usually a good place to start. They can rule out quite a few things quickly.
2. We are using a combo of Meraki MX90 and 2 MX60's for FW, QOS and security.
3. Replacing the Linksys with real Cisco anything would be a good choice. I have used many ASA devices in the past and they are all good units. As far as fixing the issue,
 
Ping is pretty useless. See where you are losing packets. Use MTR. If your linksys is dropping shit on the floor, it's silly to blame your T1.
 
Have you tried connecting a device directly to the DSU/TSU?

Those routers are older than dinosaur dung and are pieces of shit. I'd replace that with a Fortinet or ASA or even SRX. If budget is tight then maybe a EdgeRouter Lite.


Any connection will experience high latency when you saturate the upload. T1, Fiber, etc. Doesn't matter. ICMP traffic is also low priority and can be dropped if the endpoint receiving it is overloaded.
 
Last edited:
You probably have several different issues which you lump together as one...

#1
First of all, a T1 connection gives you ~1.5mbit/s both ways if it's working with its full capacity. It works like any other connection that's full duplex like a fiber connection. The main difference is the speed or the amount of lanes theoretically speaking (I'll get to that futher down).. If you put 20 devices on a T1 it will be slower than if 2 are using it, same with fiber but higher numbers.

What you have to understand is that you can not control inbound bandwidth reliably. To give an example..

The main road in town X has two lanes in both directions. If lets say 20-40 ppl used each lane during rush hour it would be fine however if your neighboring town also decides to go there at the same time you end up with a traffic jam to town. This pretty much happens with your inbound traffic, the two options you have is to have a traffic police (your gateway/firewall) to tell the extra visitors to go home directly before entering or tell them to line up and pray that they don't lose interest and go home (ie timeout). Adding to this is that your traffic police is stupid so he can't tell if some ppl are more important than others (ie not able to prioritize).

The good thing is though that he is however able to direct traffic so everyone who wants to get out of town does so rather timely and VIPs gets prioritized (QoS/shaping) or doesn't give a shit (ie, no QoS/shaping at all).

#2
You can use anything that been made the last 4 years without any issues, the question is what kind of features do you need.

#3
Maybe/Maybe not depending on how the Linksys is setup and your users.

#4
No, idea... Anything that runs OpenWRT/Linux does have CoDeL support. pfsense have ported it to their distro but I don't know how well it works.

So, to sum it all up.

Ping is not bad, but it doesn't really show anything when your connection is saturated inbound. It can possibly how overloaded it is but its not a very reliable way. Ping is however a very good way to test if your outbound shaping is working as intended (implied that ping is prioritized) as in if inbound isn't saturated but outbound is. What you can however do is limit the incoming type of traffic by blocking requests for it. What this means is limiting type of data your users can access.

First you kill off everything that isn't essential like music streaming, p2p etc (port blocking). Then you try to sanitize traffic, meaning that facebook isn't probably essential, video clips (webproxy). After you've done this your traffic has probably dropped quite a bit, if its still slow and you have 10+ users you can tell your webproxy to cache data, do note that this will only speed up things if ppl are visiting the same pages frequently.

I honestly doubt that you're Linksys is too slow to handle this kind of traffic, and there a few things to check. Are all non necessary services blocked (ports blocked)? Has anyone actually looked at how fast your connection really is? What I mean by this is you need to tell your router what kind of bandwidth you actually have, yours might actually only do 1mbit instead of 1.5mbit for instance which makes QoS (if enabled) useless if incorrectly set. That said, QoS/CoDel will not magically make your connection super fast if you have too many clients competing about bandwidth.

//Danne
 
1) it is probably cheaper to upgrade your T1 line to a DSL or cablemodem line. T1s are darn expensive on a monthly basis - and the modern internet doesn't run well at 1.44mbits/sec anymore.

2) Either the linksys or the T1 interface box could be at fault. If the T1 is a cisco router, RED is probably available to turn on, but it doesn't do you much good on the modern internet. WFQ if available, would help somewhat.

3) In terms of "fixing" things on the T1 today, you could replace the linksys box with anything that runs dd-wrt, openwrt barrier breaker, or cerowrt, (ipfire, pfsense, and a few others have fq_codel also) and turn on their QoS with a rate slightly below what the T1 provides. In all those OSes there is a modern fair queuing + aqm + prioritization system that will do wonders (and would also help on the DSL or cablemodem upgrade path)

We use the netperf-wrapper tool to get a good grip on network behavior these days:

cable: http://snapon.lab.bufferbloat.net/~cero2/jimreisert/results.html
adsl: http://burntchrome.blogspot.com/2014/03/cerowrt-31032-12-sqm-comparison-on.html
 
QoS does wonder on anything that's asymmetric in general...
//Danne
 
You must log in or register to reply here.
Back
Top