Description

This article explains one possible root cause for high CPU utilization in the Routing-engine (RE) of a SRX firewall when the following processes are also displaying CPU usage:

swi2: netisr 0
irq10: em0 em1+++*"

Symptoms

Idle cpu is low and Interrupt and Kernel are high: 

root@srx> show chassis routing-engine

Routing Engine status:
Slot 0:
Current state                  Master
Election priority              Master (default)
Temperature                 28 degrees C / 82 degrees F
CPU temperature             23 degrees C / 73 degrees F
DRAM                      3313 MB (16384 MB installed)
Memory utilization          16 percent
CPU utilization:
User                       5 percent
Background                 0 percent
Kernel                    30 percent   
Interrupt                 30 percent    
Idle                      35 percent    
Model                          RE-S-1800x4
Serial ID                      0000000000
Start time                     2020-05-05 04:00:56 UTC
Uptime                         8 days, 12 hours, 49 minutes, 38 seconds
Last reboot reason             0x1:power cycle/failure
Load averages:                 1 minute   5 minute  15 minute
22.42 16.35      14.46

Processes: 
root@srx> show system processes extensive | except 0.0

last pid: 19437;  load averages: 22.42, 16.35, 14.46  up 8+12:50:10    16:50:36
193 processes: 3 running, 175 sleeping, 1 zombie, 14 waiting

Mem: 318M Active, 101M Inact, 159M Wired, 1016M Cache, 69M Buf, 1644M Free
Swap: 3313M Total, 3313M Free

PID USERNAME      THR PRI NICE   SIZE    RES STATE    TIME WCPU COMMAND
11 root            1 171   52     0K    12K RUN     84.7H 35.21% idle
14 root            1 -40 -159     0K    12K WAIT    29.3H 12.26% swi2: netisr 0   
20 root            1 -68 -187     0K    12K RUN     20.5H 11.28% irq11: em0 em1 em*  
2643 root            1 132    0   125M 25532K select 983:34 6.93% chassisd
18965 root            1 121    0 33840K 26748K select   0:37 2.78% mgd
12 root            1 -20 -139     0K    12K WAIT   336:52 1.95% swi7: clock sio
19394 root            1  98    0 20624K 15088K select   0:00 1.54% cli

Solution

The swi2: netisr 0 process is a software interrupt handling thread and it is responsible for RE network traffic handling (RE-RE, RE-PFES connections, etc)​. ARP packets are sent to the RE for handling via the "swi2: netisr 0"  process and a high ARP activity in the network can cause the process CPU usage to go high.

The following could be indicators of an ARP storm / L2 loop in the network:
 

  • Prior JunOS release 17.3:
root@srx> show system queues
[output truncated]
input protocol              bytes          max  packets max    drops
splfwdq                         0      1000000        0 1000        0
splnetq                         0      1000000        0 1000        0
arpintrq                        0         3000        0 50     5419402
  • From JunOS release 17.3+:
user@srx> show system queues
[output truncated]
input protocol              bytes          max  packets      max    drops
splfwdq                         0      1000000        0     1000        0
splnetq                         0      1000000        0     1000        0
tnpintrq                        0      1250000        0     4166        5419402


ARP activity can be monitored with the following Shell and CLI commands:

  • Shell
user@srx> start shell
user@srx:~ # sysctl net.netdaemon_stats | egrep arp
Netisr arp handler      0 calls
  • CLI
root@srx> show system statistics arp
arp:
[output truncated]
482411627 datagrams received
4978470 ARP requests received
316177 ARP replies received
29134 resolution request received
402768 received proxy requests
7509132 datagrams with my own hardware address
34289 datagrams for an address not on the interface
1972855 datagrams with source address duplicate to mine
467197936 datagrams which were not for me
359441 packets discarded waiting for resolution
39 packets sent after waiting for resolution
3733785 ARP requests sent
5381238 ARP replies sent

1. Confirm the active interfaces in the device:

root@srx> show interfaces extensive | match "Physical interface" | match "Physical link is Up"
      Physical interface: reth1  , Enabled, Physical link is Up 
      Physical interface: reth3  , Enabled, Physical link is Up
      Physical interface: reth7  , Enabled, Physical link is Up

2. Run "monitor traffic interface" on these interfaces for 10 seconds and confirm the total number of captured packets. Note this command only capture packets source or destined to the RE. 
root@srx> show interfaces terse | match reth1
reth1.0              up    up   inet     10.10.10.1/24


root@srx> monitor traffic interface reth1 no-resolve

[output truncated]
18:34:53.741134  In arp who-has 10.10.10.43 tell 10.10.10.1
18:34:53.741181  In arp who-has 10.10.10.163 tell 10.10.10.1
18:34:53.741193  In arp who-has 10.10.10.195 tell 10.10.10.1
18:34:53.742219  In arp who-has 10.10.10.54 tell 10.10.10.1
18:34:53.742237  In arp who-has 10.10.10.51 tell 10.10.10.1
18:34:53.842099  In arp who-has 10.10.10.144 tell 10.10.10.1
18:34:53.842109  In arp who-has 10.10.10.178 tell 10.10.10.1
18:34:53.843184  In arp who-has 10.10.10.175 tell 10.10.10.1​
[output truncated]
20955 packets received by filter
0 packets dropped by kernel
A high amount of packets, captured in 10 seconds, can be seen above and these have to be processed by the RE. Also ingress ARP requests with the source address of reth1.0 are noticed, meaning that these packets were sent from the SRX and looped back to the device. The L2 network has to be investigated because there is a network loop.

Modification History

2021-01-06: Added new output for "show system queues" for Junos releases 17.3+. Also added a shell command for checking for high ARP activity.

Related Information