cisco guru's. need some help.. high cpu utilization on 2611xm router

C

cyr0n_k0r

Supreme [H]ardness
Joined
Mar 30, 2001
Messages
5,360
here is a show proc cpu during max load and normal load

MAX LOAD
router#
router#
router#show proc cpu
CPU utilization for five seconds: 99%/50%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 0 1 0 0.00% 0.00% 0.00% 0 Chunk Manager

2 328563 639139 514 0.07% 0.01% 0.00% 0 Load Meter

3 941 268 3511 0.07% 0.00% 0.03% 0 Exec

4 12233 26630 459 0.00% 0.00% 0.00% 0 DHCPD Timer

5 4070062 401319 10141 0.30% 0.10% 0.06% 0 Check heaps

6 3968 984 4032 0.00% 0.00% 0.00% 0 Pool Manager

7 0 2 0 0.00% 0.00% 0.00% 0 Timers

8 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun

9 4 2 2000 0.00% 0.00% 0.00% 0 AAA high-capacit

10 14859 106494 139 0.00% 0.00% 0.00% 0 Environmental mo

11 149971 121796 1231 0.00% 0.00% 0.00% 0 ARP Input

12 90972 159729 569 0.00% 0.00% 0.00% 0 HC Counter Timer

13 0 2 0 0.00% 0.00% 0.00% 0 DDR Timers

14 4 2 2000 0.00% 0.00% 0.00% 0 Entity MIB API

15 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect

16 611194 3195145 191 0.00% 0.00% 0.00% 0 GraphIt

17 0 2 0 0.00% 0.00% 0.00% 0 Dialer event

18 0 2 0 0.00% 0.00% 0.00% 0 SMART

19 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd

20 502348 598836 838 0.00% 0.00% 0.00% 0 Net Background

21 8 95 84 0.00% 0.00% 0.00% 0 Logger

22 785459 3172604 247 0.00% 0.00% 0.00% 0 TTY Background

23 1314844 3195160 411 0.07% 0.01% 0.00% 0 Per-Second Jobs

24 0 2 0 0.00% 0.00% 0.00% 0 SM Monitor

25 0 1 0 0.00% 0.00% 0.00% 0 dev_device_inser

26 0 1 0 0.00% 0.00% 0.00% 0 dev_device_remov

27 0 1 0 0.00% 0.00% 0.00% 0 Net Input

28 292219 639142 457 0.00% 0.00% 0.00% 0 Compute load avg

29 2999423 53618 55940 0.00% 0.04% 0.00% 0 Per-minute Jobs

30 0 2 0 0.00% 0.00% 0.00% 0 AAA Server

31 0 1 0 0.00% 0.00% 0.00% 0 AAA ACCT Proc

32 0 1 0 0.00% 0.00% 0.00% 0 ACCT Periodic Pr

33 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT

34 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R

35 61267103 10648252 5753 41.32% 46.47% 47.59% 0 IP Input

36 0 1 0 0.00% 0.00% 0.00% 0 ICMP event handl

37 427564 407138 1050 0.00% 0.01% 0.00% 0 CDP Protocol

38 4 3 1333 0.00% 0.00% 0.00% 0 PPP Hooks

39 0 1 0 0.00% 0.00% 0.00% 0 X.25 Encaps Mana

40 1963 15405 127 0.00% 0.00% 0.00% 0 TCP Timer

41 232 50 4640 0.00% 0.00% 0.00% 0 TCP Protocols

42 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers

43 2832 10654 265 0.00% 0.00% 0.00% 0 HTTP CORE

44 0 2 0 0.00% 0.00% 0.00% 0 PPP IP Route

45 0 2 0 0.00% 0.00% 0.00% 0 PPP IPCP

46 497340 63931 7779 0.00% 0.00% 0.00% 0 IP Background

47 20663 53252 388 0.00% 0.00% 0.00% 0 IP RIB Update

48 125200 53254 2350 0.00% 0.00% 0.00% 0 Adj Manager

49 3614638 5249923 688 0.00% 0.00% 0.00% 0 CEF process

50 0 1 0 0.00% 0.00% 0.00% 0 Asy FS Helper

51 0 1 0 0.00% 0.00% 0.00% 0 RARP Input

52 136 77 1766 0.00% 0.00% 0.00% 0 DHCPD Receive

53 650286 53230 12216 0.00% 0.05% 0.05% 0 IP Cache Ager

54 0 1 0 0.00% 0.00% 0.00% 0 COPS

55 0 2 0 0.00% 0.00% 0.00% 0 PPP Bind

56 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall

57 0 2 0 0.00% 0.00% 0.00% 0 X.25 Background

58 2832071 3382783 837 1.36% 1.13% 1.18% 0 Inspect Timer

59 27736 110805 250 0.00% 0.00% 0.00% 0 CEF Scanner

60 0 2 0 0.00% 0.00% 0.00% 0 URL filter proc

61 2466 10653 231 0.00% 0.00% 0.00% 0 Authentication P

62 0 1 0 0.00% 0.00% 0.00% 0 Auth-proxy AAA B

63 0 1 0 0.00% 0.00% 0.00% 0 IDS Timer

64 31147 127831 243 0.00% 0.00% 0.00% 0 CRM_CALL_UPDATE_

65 0 2 0 0.00% 0.00% 0.00% 0 LOCAL AAA

66 0 2 0 0.00% 0.00% 0.00% 0 ENABLE AAA

67 0 2 0 0.00% 0.00% 0.00% 0 LINE AAA

68 4 2 2000 0.00% 0.00% 0.00% 0 TPLUS

69 0 1 0 0.00% 0.00% 0.00% 0 Key chain liveke

71 0 2 0 0.00% 0.00% 0.00% 0 AAA SEND STOP EV

72 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps

74 4177596 6345017 658 0.75% 0.73% 0.80% 0 IP NAT Ager

75 41607 904730 45 0.00% 0.00% 0.00% 0 DHCPD Database
router#
router#
Click to expand...

NORMAL LOAD
router>show proc cpu
CPU utilization for five seconds: 50%/48%; one minute: 53%; five minutes: 79%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 0 1 0 0.00% 0.00% 0.00% 0 Chunk Manager

2 328707 639458 514 0.00% 0.00% 0.00% 0 Load Meter

3 212 58 3655 0.07% 0.03% 0.00% 0 Exec

4 12237 26643 459 0.00% 0.00% 0.00% 0 DHCPD Timer

5 4072710 401472 10144 0.00% 0.09% 0.06% 0 Check heaps

6 3972 986 4028 0.00% 0.00% 0.00% 0 Pool Manager

7 0 2 0 0.00% 0.00% 0.00% 0 Timers

8 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun

9 4 2 2000 0.00% 0.00% 0.00% 0 AAA high-capacit

10 14867 106547 139 0.00% 0.00% 0.00% 0 Environmental mo

11 150047 121853 1231 0.00% 0.00% 0.00% 0 ARP Input

12 91004 159808 569 0.00% 0.00% 0.00% 0 HC Counter Timer

13 0 2 0 0.00% 0.00% 0.00% 0 DDR Timers

14 4 2 2000 0.00% 0.00% 0.00% 0 Entity MIB API

15 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect

16 611445 3196736 191 0.00% 0.00% 0.00% 0 GraphIt

17 0 2 0 0.00% 0.00% 0.00% 0 Dialer event

18 0 2 0 0.00% 0.00% 0.00% 0 SMART

19 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd

20 502666 599149 838 0.00% 0.00% 0.00% 0 Net Background

21 8 95 84 0.00% 0.00% 0.00% 0 Logger

22 785779 3173887 247 0.00% 0.00% 0.00% 0 TTY Background

23 1315564 3196759 411 0.00% 0.00% 0.00% 0 Per-Second Jobs

24 0 2 0 0.00% 0.00% 0.00% 0 SM Monitor

25 0 1 0 0.00% 0.00% 0.00% 0 dev_device_inser

26 0 1 0 0.00% 0.00% 0.00% 0 dev_device_remov

27 0 1 0 0.00% 0.00% 0.00% 0 Net Input

28 292379 639461 457 0.00% 0.00% 0.00% 0 Compute load avg

29 3000891 53646 55938 0.00% 0.03% 0.00% 0 Per-minute Jobs

30 0 2 0 0.00% 0.00% 0.00% 0 AAA Server

31 0 1 0 0.00% 0.00% 0.00% 0 AAA ACCT Proc

32 0 1 0 0.00% 0.00% 0.00% 0 ACCT Periodic Pr

33 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT

34 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R

35 62637785 10652123 5880 0.95% 6.32% 29.38% 0 IP Input

36 0 1 0 0.00% 0.00% 0.00% 0 ICMP event handl

37 427753 407336 1050 0.00% 0.00% 0.00% 0 CDP Protocol

38 4 3 1333 0.00% 0.00% 0.00% 0 PPP Hooks

39 0 1 0 0.00% 0.00% 0.00% 0 X.25 Encaps Mana

40 1963 15405 127 0.00% 0.00% 0.00% 0 TCP Timer

41 232 50 4640 0.00% 0.00% 0.00% 0 TCP Protocols

42 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers

43 2832 10659 265 0.00% 0.00% 0.00% 0 HTTP CORE

44 0 2 0 0.00% 0.00% 0.00% 0 PPP IP Route

45 0 2 0 0.00% 0.00% 0.00% 0 PPP IPCP

46 497620 63957 7780 0.00% 0.00% 0.00% 0 IP Background

47 20679 53278 388 0.00% 0.00% 0.00% 0 IP RIB Update

48 125256 53280 2350 0.00% 0.00% 0.00% 0 Adj Manager

49 3615071 5250410 688 0.00% 0.02% 0.00% 0 CEF process

50 0 1 0 0.00% 0.00% 0.00% 0 Asy FS Helper

51 0 1 0 0.00% 0.00% 0.00% 0 RARP Input

52 136 77 1766 0.00% 0.00% 0.00% 0 DHCPD Receive

53 653130 53255 12264 0.00% 0.05% 0.05% 0 IP Cache Ager

54 0 1 0 0.00% 0.00% 0.00% 0 COPS

55 0 2 0 0.00% 0.00% 0.00% 0 PPP Bind

56 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall

57 0 2 0 0.00% 0.00% 0.00% 0 X.25 Background

58 2857249 3386357 843 0.00% 0.03% 0.44% 0 Inspect Timer

59 27748 110853 250 0.00% 0.00% 0.00% 0 CEF Scanner

60 0 2 0 0.00% 0.00% 0.00% 0 URL filter proc

61 2470 10658 231 0.00% 0.00% 0.00% 0 Authentication P

62 0 1 0 0.00% 0.00% 0.00% 0 Auth-proxy AAA B

63 0 1 0 0.00% 0.00% 0.00% 0 IDS Timer

64 31183 127894 243 0.00% 0.00% 0.00% 0 CRM_CALL_UPDATE_

65 0 2 0 0.00% 0.00% 0.00% 0 LOCAL AAA

66 0 2 0 0.00% 0.00% 0.00% 0 ENABLE AAA

67 0 2 0 0.00% 0.00% 0.00% 0 LINE AAA

68 4 2 2000 0.00% 0.00% 0.00% 0 TPLUS

69 0 1 0 0.00% 0.00% 0.00% 0 Key chain liveke

71 0 2 0 0.00% 0.00% 0.00% 0 AAA SEND STOP EV

72 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps

74 4195783 6349368 660 0.07% 0.04% 0.27% 0 IP NAT Ager

75 41627 905138 45 0.00% 0.00% 0.00% 0 DHCPD Database
router>
router>
Click to expand...

The only problem is, I can't figure out what is causing the CPU to max out. The traffic isn't changing, and I can't identify what is causing the problem. I have however narrowed it down to PID 35 "IP Input", whatever that does.

I've disabled logging, debugging, etc.
 
IP input is exactly like it sounds.... the processing of IP packets that are received then delivered by the switch. Your router seems to be process(software) switching everything, which is not too efficient. Route maps applied to outbound interfaces? Routing on stick at all? How about posting up a show run while its under load.

What I would really want to see is a show tech-support, but dont bother flooding this thread with that, heh.
 
router#sh ru
Building configuration...

Current configuration : 5223 bytes
!
! Last configuration change at 19:33:58 MST Mon Dec 10 2007
! NVRAM config last updated at 11:34:03 MST Tue Dec 4 2007
!
version 12.3
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname router
!
boot-start-marker
boot-end-marker
!
no logging on
enable secret 5 **********
!
clock timezone MST -7
no network-clock-participate slot 1
no network-clock-participate wic 0
no aaa new-model
ip subnet-zero
ip cef
!
!
ip name-server **********
!
ip inspect max-incomplete high 1100
ip inspect max-incomplete low 900
ip inspect one-minute high 1100
ip inspect one-minute low 900
ip inspect udp idle-time 60
ip inspect dns-timeout 10
ip inspect name WAN http timeout 3600
ip inspect name WAN ftp timeout 3600
ip inspect name WAN tftp timeout 3600
ip inspect name WAN udp timeout 15
ip inspect name WAN tcp timeout 3600
ip audit po max-events 100
!
!
!
!
!
interface FastEthernet0/0
description wan
ip address ********** 255.255.255.252
ip nat outside
ip inspect WAN in
duplex auto
speed auto
!
interface FastEthernet0/1
description lan
ip address 10.1.1.1 255.255.255.0
ip nat inside
duplex auto
speed auto
!
ip default-gateway **********
ip nat inside source route-map NAT interface FastEthernet0/0 overload
ip nat inside source static tcp 10.1.1.5 80 ********** 80 extendable
ip nat inside source static tcp 10.1.1.5 443 ********** 443 extendable
ip nat inside source static tcp 10.1.1.5 3389 ********** 3389 extendable
ip nat inside source static tcp 10.1.1.6 80 ********** 80 extendable
ip nat inside source static tcp 10.1.1.6 443 ********** 443 extendable
ip nat inside source static tcp 10.1.1.6 3389 ********** 3389 extendable
ip nat inside source static tcp 10.1.1.7 25 ********** 25 extendable
ip nat inside source static tcp 10.1.1.7 80 ********** 80 extendable
ip nat inside source static tcp 10.1.1.7 3389 ********** 3389 extendable
ip nat inside source static tcp 10.1.1.5 9500 ********** 9500 extendable
ip nat inside source static tcp 10.1.1.5 9501 ********** 9501 extendable
ip nat inside source static tcp 10.1.1.5 9502 ********** 9502 extendable
ip nat inside source static tcp 10.1.1.5 9503 ********** 9503 extendable
ip nat inside source static tcp 10.1.1.5 9504 ********** 9504 extendable
ip nat inside source static tcp 10.1.1.5 9505 ********** 9505 extendable
ip nat inside source static tcp 10.1.1.5 9506 ********** 9506 extendable
ip nat inside source static tcp 10.1.1.5 9507 ********** 9507 extendable
ip nat inside source static tcp 10.1.1.5 9508 ********** 9508 extendable
ip nat inside source static tcp 10.1.1.5 9509 ********** 9509 extendable
ip nat inside source static tcp 10.1.1.5 9510 ********** 9510 extendable
ip nat inside source static tcp 10.1.1.5 9511 ********** 9511 extendable
ip nat inside source static tcp 10.1.1.5 9512 ********** 9512 extendable
ip nat inside source static tcp 10.1.1.5 9513 ********** 9513 extendable
ip nat inside source static tcp 10.1.1.5 9514 ********** 9514 extendable
ip nat inside source static tcp 10.1.1.5 9515 ********** 9515 extendable
ip nat inside source static tcp 10.1.1.5 9516 ********** 9516 extendable
ip nat inside source static tcp 10.1.1.5 9517 ********** 9517 extendable
ip nat inside source static tcp 10.1.1.5 9518 ********** 9518 extendable
ip nat inside source static tcp 10.1.1.5 9519 ********** 9519 extendable
ip nat inside source static tcp 10.1.1.5 9520 ********** 9520 extendable
ip nat inside source static tcp 10.1.1.5 9521 ********** 9521 extendable
ip nat inside source static tcp 10.1.1.5 9522 ********** 9522 extendable
ip nat inside source static tcp 10.1.1.5 9523 ********** 9523 extendable
ip nat inside source static tcp 10.1.1.5 9524 ********** 9524 extendable
ip nat inside source static tcp 10.1.1.5 9525 ********** 9525 extendable
ip nat inside source static tcp 10.1.1.5 9526 ********** 9526 extendable
ip nat inside source static tcp 10.1.1.5 9527 ********** 9527 extendable
ip nat inside source static tcp 10.1.1.5 9528 ********** 9528 extendable
ip nat inside source static tcp 10.1.1.5 9529 ********** 9529 extendable
ip nat inside source static tcp 10.1.1.5 9530 ********** 9530 extendable
ip nat inside source static tcp 10.1.1.21 80 ********** 80 extendable
ip nat inside source static tcp 10.1.1.21 443 ********** 443 extendable
ip nat inside source static tcp 10.1.1.21 3389 ********** 3389 extendable
ip nat inside source static tcp 10.1.1.20 80 ********** 80 extendable
ip nat inside source static tcp 10.1.1.20 443 ********** 443 extendable
ip nat inside source static tcp 10.1.1.7 8080 ********** 8080 extendable
ip nat inside source static tcp 10.1.1.22 80 ********** 80 extendable
ip nat inside source static tcp 10.1.1.22 443 ********** 443 extendable
no ip http server
no ip classless
ip route 0.0.0.0 0.0.0.0 **********
!
!
access-list 110 permit ip 10.1.1.0 0.0.0.255 any
route-map NAT permit 10
match ip address 110
!
!
line con 0
line aux 0
line vty 0 4
password **********
login
!
!
end

router#
Click to expand...
ip cef IS enabled.
 
Yep, you have a route-map :) Generally when a lot of traffic traverses the route-map it will begin to process switch that traffic. What's weird is that you do have cef enabled and by default this is the switching type used with pbr since you have 12.x ios:confused: hmmmm......

I would try and enable fast-switching on the interface for a quick test to see if you see a performance increase. Also, there is no reason you should be using that route-map, you can achieve the same results using a standard source list entry. I would change your PAT statement accordingly and see if you router begins to use to cef switch everything.
 
doing a "show proc cpu sorted | exc 0.00" will give you much cleaner output in finding cpu hogging processes.

Your CPU's interrupt usage is staying at 50% on both examples, which means the router is doing quite a bit of network IO. According to Cisco, a 2611xm can only route 7.68Mbps of traffic (in an ideal situation; "Numbers are given with 64 byte packet size, IP only, and are only an indication of raw switching performance. These are testing numbers, usually with FE to FE or POS to POS, no services enabled. As you add ACL's,
encryption, compression, etc - performance will decline significantly from the given numbers, unless it is a hardware-assisted platform")

How hard are you working that thing?
 
router#sh proc cpu sorted | exc 0.00
CPU utilization for five seconds: 47%/46%; one minute: 48%; five minutes: 46%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
35 68044298 10760288 6323 0.63% 0.44% 0.33% 0 IP Input

3 152 105 1447 0.15% 0.08% 0.02% 0 Exec

router#
Click to expand...

We push about 10-20Mbps through it all day and night.

We WERE running a proxy server through it, but I disabled that and am not seeing any more cpu spikes. Could that have been the problem? I am going to continue to monitor things.
 
You must log in or register to reply here.
Back
Top