Description

We might see 'kernel: interrupt storm detected on "irq11:"; throttling interrupt source' log errors on the routing-engine that leads to CPU hog, causing routing protocols to flap on the worst case. 

Symptoms

On the MX Devices with Next generation RE's, we might see the below log errors throttling at certain instances, causing the CPU hog.

kernel: interrupt storm detected on "irq11:"; throttling interrupt source

kernel: interrupt storm detected on "irq11:"; throttling interrupt source

When the issue occurs, the high rate of irq11 (such as 12K rate on irq11 from acb0) could be seen:

root@device> show system virtual-memory no-forwarding

<..>

interrupt             total   rate

irq4: uart0 uart2         1964     0

irq11: ichsmb0 acb0    192406873490   12918

irq14: ata0            33510     0

cpu0:timer         12967935126    870

irq256: +           14892331     0

irq257: ixlv0:que 0     2759115969    185

irq258: igb0:que 0      1060657961    71

irq264: virtio_pci1      11600503     0

irq265: virtio_pci1         155     0

irq267: virtio_pci2       196328     0

irq269: virtio_pci3       3650754     0

irq273: virtio_pci4         255     0

cpu1:timer          1787726319    120

cpu3:timer          1776602530    119

cpu2:timer          1784730982    119

Total           214574018177   14407

 

And the system process also shows there is CPU high for intr process.

root@device> show system processes extensive no-forwarding

 

last pid: 9229; load averages: 0.83, 0.82, 0.76 up 172+09:02:11  06:43:20

167 processes: 2 running, 164 sleeping, 1 waiting

 

Mem: 2034M Active, 16G Inact, 1670M Wired, 1651M Buf, 27G Free

Swap: 3072M Total, 3072M Free

 

 PID USERNAME   THR PRI NICE SIZE  RES STATE C TIME  WCPU COMMAND

  10 root      4 155 ki31  0K  64K CPU3  3  ??? 332.86% idle

  11 root      43 -72  -  0K 688K WAIT 255 1825.6 54.88% intr

The exact trigger of this issue is not known, and it is tracked on the PR1386306.

What could be impacted:

When we see the interrupts and routing-engine errors occurring frequently on the device, then it is not suggested to execute any commands that spikes the CPU more.

 

Some of the below actions are to be restricted, when we see interrupts throttling on the RE:

  1. Configuring trace-options and sampling with higher rates.
  2. Creation of VMHOST snapshots, could increase the threshold limit of an RE.
  3. Configuring Port mirroring on a scaled network environment.
  4. Executing commit full or Multiple commits or changes related to routing-protocols.

The above action could cause CPU hog, and it may result in the routing-protocol to flap on worst cases.

Solution

The below mentioned action plan has to be performed during the MW time.

 

Step 1: Kill the ‘intr’ process on the master RE.

 

> start shell user root

Password:

root@MX1:~ # kill -9 <pid>

 

 Example:

> show system processes extensive | match intr

PID USERNAME  PRI NICE SIZE  RES STATE  C TIME  WCPU COMMAND

33134 root    155 ki31  0B  80K CPU0  0 22.3H 99.37% intr

 

> start shell user root

Password:

root@MX1:~ # kill -9 33134

Step 2: If the issue still persist, then you may need to reboot the RE.

Modification History

2025-02-06 : Article Created
2025-07-30: Adding MX480 to the categories