Slow upload on Cisco 2800

S

slowbiznatch

Gawd
Joined
Oct 24, 2001
Messages
856
I'm using a Cisco 2801 with 4 bonded T1 connections. The T1s terminate to an Adtran router provided by my ISP (Paetec). If I connect directly to the outside port on the Adtran, I get perfect speeds both ways. My download speeds while connected with my Cisco are hovering around 5600 kbps, but my upload speed seems to be maxing out at about 1500 kbps. Here's a look at the outside interface config:

Code: 
interface FastEthernet0/0
 description OUTSIDE INTERFACE$FW_OUTSIDE$$ETH-WAN$
 ip address 64.xxx.xxx.xxx 255.255.255.192
 ip access-group Inbound in
 ip mtu 1400
 ip inspect InspectOutbound out
 ip flow ingress
 ip nat outside
 ip virtual-reassembly
 no ip route-cache cef
 no ip route-cache
 speed auto
 full-duplex
 no cdp enable
 no mop enabled
 crypto map VPN
end

And here's the status:
Code: 
FastEthernet0/0 is up, line protocol is up
  Hardware is Gt96k FE, address is 0017.5abb.ccb0 (bia 0017.5abb.ccb0)
  Description: OUTSIDE INTERFACE$FW_OUTSIDE$$ETH-WAN$
  Internet address is 64.1xxx.xxx.xxx/26
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, 100BaseTX/FX
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:05, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/42/0 (size/max/drops/flushes); Total output drops: 43
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 455000 bits/sec, 45 packets/sec
  5 minute output rate 60000 bits/sec, 23 packets/sec
     410897 packets input, 225051540 bytes
     Received 228 broadcasts, 0 runts, 0 giants, 0 throttles
     1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog
     0 input packets with dribble condition detected
     248056 packets output, 63250096 bytes, 0 underruns
     0 output errors, 0 collisions, 8 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

I know enough about our setup to get by, but I am by no means a Cisco Engineer. Anyone want to take a stab at my problem? If you need anymore information, feel free to PM me or ask in the thread.

Thank you.
 
tried enabling CEF routing? Could also be the doubling nating. What are you using to test the U/L?
 
I'm not familiar with CEF routing, and I haven't tried it.

I'm pretty sure I'm not double NATing. The outside interface on the Adtran router has no firewall or NAT enabled.

To test the upload speed, I'm using speakeasy.net, speedtest.net, or any of the dslreports tests.
 
Oh one thing to note. On my 2611XM it made my router really unstable. So if you have stability issues disable it and continue looking, but it really did improve the performance when it wasn't rebooting from a crash.
 
Do a "sh cpu proc hist". Or, when you're running your test, "sh proc cpu sort". If you're not running cef or fast switching, you'll see huge CPU spikes because everything is being process switched (all software).

Furthermore, looking at the stats below, it looks like when your router is not running cef, it is limited to 3,000 pps and 1.536Mbps, which is right around what you're hitting. With CEF enabled (global command: ip cef), you can push that up to ~45Mb, which is well above 4xT1.

Cisco likes to make this hard to find on their website for some reason. I haven't seen this in a while.. never knew my 7600s were THAT good haha.
http://www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf
 
Should I experience any downtime if I enable CEF? I guess I should have tested it last night...

This router terminates about 14 IPSEC DMVPN tunnels, so I'd rather not let those go down if I don't have to.
 
just2cool said:
Do a "sh cpu proc hist". Or, when you're running your test, "sh proc cpu sort". If you're not running cef or fast switching, you'll see huge CPU spikes because everything is being process switched (all software).
Click to expand...

During my upload test I don't see any CPU spikes above 30%.
 
Actually, I just looked through the config this morning and CEF IS enabled.

Any idea what else I should be looking for?
 
CEF may be enabled globally, but it doesn't appear to be enabled on the interfaces:

Code: 
 no ip route-cache cef
 no ip route-cache
 
I enabled CEF on the interface, but it still seems to be doing the same thing. Here's some info from CEF:

Code: 
kcmo-2801#sho ip cef summary
IP CEF with switching (Table Version 166), flags=0x0
  165 routes, 0 reresolve, 0 unresolved (0 old, 0 new), peak 3
  3 instant recursive resolutions, 0 used background process
  165 leaves, 23 nodes, 49000 bytes, 179 inserts, 14 invalidations
  0 load sharing elements, 0 bytes, 0 references
  universal per-destination load sharing algorithm, id 9923FA95
  3(0) CEF resets, 1 revisions of existing leaves
  Resolution Timer: Exponential (currently 1s, peak 1s)
  1 in-place/0 aborted modifications
  refcounts:  6771 leaf, 6144 node

  Table epoch: 0 (165 entries at this epoch)

Adjacency Table has 106 adjacencies

Also, if I do sh ip cef I can see it working on interface FastEthernet0/0...
 
Ok, now that CEF is building adjacencies, there's no reason why that router can't route up to at least 40Mbps.

So, I think the problem has something to do with the Adtran router and the way it's load balacing. It's acting right now like it's hashing based on source/destination, not on a per packet basis.

If I were you, I would run a concurrent speed test on 2 machines. If they both get 1.5Mbps, then the Adran is utilizing 2 out of the 4 circuits and you really don't have to worry about the speedtest only reporting 1.5 back to you. However, if they're getting a combined total of 1.5, then it definitely has something to do with the load balancing/NAT.. are you using one "inside global" address in your config?

Test this out and let us know.
 
I can get aggregate upload bandwidth across 3 different workstations that is greater than 3mb, so I think we're on to something.

What do you mean by "are you using one "inside global" address in your config?" Which config are you talking about? The Adtran or the Cisco? We don't maintain the Adtran, but I should be able to work with Paetec easily if I can identify a problem.
 
Cisco config.

Here's a 3 second summary on Cisco NAT:
inside local = private address space
inside global = outside address(es) assigned to you
outside global = host you are communicating to on the outside
outside local = translating one of your "inside locals" to an "outside global".. don't worry about this one too much.

If you don't know what I'm talking about, you probably only have 1 inside global (public IP address). Is that the case?

I think this has something to do with the way NAT is affecting the hashing algorithm on the Adtran router.
 
Alright. I'm going to try and guess your topology.

So, do you have a /30 for the Cisco<->Adtran link and a /28 that sits behind the Cisco router? If so, run the speedtest on one of the /28 hosts.

I'm thinking that any hosts with public addresses will be able to get >1.5Mbps and any NATed addresses will be capped a 1.5 agg due to inefficient load balancing.

Either way, you should bring this up with your ISP and have them look at the Adtran.
 
You must log in or register to reply here.
Back
Top