This article explains how the high CPU events can cause the routes to stuck on the KRT queue and causing congestion on the device.

When high CPU events occurs and routes are stuck on the queue, we would see similar outputs:

> show krt queue 

Routing table add queue: 1 queued

 ADD table index 7, gf 2

 kqp '0x4a061300'

Interface add/delete/change queue: 0 queued

Top-priority deletion queue: 0 queued

Top-priority change queue: 0 queued

Top-priority add queue: 0 queued

high priority V4oV6 tcnh delete queue: 0 queued

high prioriy anchor gencfg delete queue: 0 queued

High-priority multicast add/change: 0 queued

Indirect next hop top priority add/change: 0 queued

Indirect next hop add/change: 8 queued

 CHANGE TO indirect to <IP address> index 1048577 (0x3b84b088)

 kqp '0xd5f38480'

>show system processes extensive no-forwarding

last pid: 75199; load averages: 1.78, 1.75, 1.70 up 215+07:06:19  22:37:15

527 threads: 6 running, 471 sleeping, 1 zombie, 49 waiting

CPU: 15.9% user, 0.0% nice, 8.4% system, 0.2% interrupt, 75.4% idle

Mem: 449M Active, 9663M Inact, 3463M Wired, 285M Buf, 80G Free

Swap: 12G Total, 12G Free

 PID USERNAME  PRI NICE SIZE  RES STATE  C TIME  WCPU COMMAND

15392 root    103  0 730M  17M CPU0  0 3568.3 100.00% eventd

15478 root    20  0 5546M 4685M kqread 2 83.1H 6.10% rpd{rpd}

> show chassis alarms

<date&time> Minor Potential slow peers are: XDPC0

'Potential slow peers are: XDPC0' alarm usually appears when the MPC10E cards experiencing route churn.  This could be caused due to the configs getting added/deleted/changed frequently or if we deactivate/activate any routing-instances or while rebooting the FPC/RE. 

Some of the possible options of events process going higher are:

• Configuring trace-options
• Configuring sampling with aggressive rate. 
  • example: configuring sampling with rate of 1000 [Which means the every thousandth packet (1 out of 1000) will be sampled here].
  • Refer: https://www.juniper.net/documentation/us/en/software/junos/cli-reference/topics/ref/statement/rate-edit-forwarding-options.html
• Configuring port-mirror on a high scaled environment.
• Configuring syslog with 'any notice' could log eve
• RE is over utilised for any unexpected events or goes faulty. 

We would say high CPU process (eventd) utilization was causing the RE to be over utilized and few routes were not been installed on to the forwarding table. This is the reason, why you have seen the routes are stuck on the KRT queue

On the above case, we could see eventd process is utilizing 100% CPU process. Eventd process is responsible for all the logging and reporting on MX and will surely overburden the RE CPU if the excessive logging is configured on the device.

Workaround:

• Try switching the mastership of an RE and reboot the affected RE.
• Try killing the process/daemon which is utilising more CPU cycles.
• Try removing trace-options, port-mirroring configurations.
• Try increase the sampling rate by referring to the technical documentation rate (Forwarding Options).
• Try removing 'any notice' statement from syslog.