We might see 'kernel: interrupt storm detected on "irq11:"; throttling interrupt source' log errors on the routing-engine that leads to CPU hog, causing routing protocols to flap on the worst case.
On the MX Devices with Next generation RE's, we might see the below log errors throttling at certain instances, causing the CPU hog.
kernel: interrupt storm detected on "irq11:"; throttling interrupt source
When the issue occurs, the high rate of irq11 (such as 12K rate on irq11 from acb0) could be seen:
root@device> show system virtual-memory no-forwarding
<..>
interrupt total rate
irq4: uart0 uart2 1964 0
irq11: ichsmb0 acb0 192406873490 12918
irq14: ata0 33510 0
cpu0:timer 12967935126 870
irq256: + 14892331 0
irq257: ixlv0:que 0 2759115969 185
irq258: igb0:que 0 1060657961 71
irq264: virtio_pci1 11600503 0
irq265: virtio_pci1 155 0
irq267: virtio_pci2 196328 0
irq269: virtio_pci3 3650754 0
irq273: virtio_pci4 255 0
cpu1:timer 1787726319 120
cpu3:timer 1776602530 119
cpu2:timer 1784730982 119
Total 214574018177 14407
And the system process also shows there is CPU high for intr process.
root@device> show system processes extensive no-forwarding
last pid: 9229; load averages: 0.83, 0.82, 0.76 up 172+09:02:11 06:43:20
167 processes: 2 running, 164 sleeping, 1 waiting
Mem: 2034M Active, 16G Inact, 1670M Wired, 1651M Buf, 27G Free
Swap: 3072M Total, 3072M Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
10 root 4 155 ki31 0K 64K CPU3 3 ??? 332.86% idle
11 root 43 -72 - 0K 688K WAIT 255 1825.6 54.88% intr
The exact trigger of this issue is not known, and it is tracked on the PR1386306.
What could be impacted:
When we see the interrupts and routing-engine errors occurring frequently on the device, then it is not suggested to execute any commands that spikes the CPU more.
Some of the below actions are to be restricted, when we see interrupts throttling on the RE:
The above action could cause CPU hog, and it may result in the routing-protocol to flap on worst cases.
The below mentioned action plan has to be performed during the MW time.
Step 1: Kill the ‘intr’ process on the master RE.
> start shell user root
Password:
root@MX1:~ # kill -9 <pid>
Example:
> show system processes extensive | match intr
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
33134 root 155 ki31 0B 80K CPU0 0 22.3H 99.37% intr
root@MX1:~ # kill -9 33134
Step 2: If the issue still persist, then you may need to reboot the RE.
2025-02-06 : Article Created2025-07-30: Adding MX480 to the categories