This Article explains the root cause and recovery of the continuous RPD core followed by protocols flaps because of rpd assertion failure seen during RPD process sync to kernel for composite nexthop [cnh] update

On QFX10002 switch with image “21.4R3-S2.3” RPD assertion failures, protocols sessions flaps and RPD cores were seen continuously as below

Assertion failure reported by RPD on krt_cnh_async.c file:

Sep 26 09:26:18  hostname rpd[80762]: JTASK_ASSERT: Assertion failed rpd[80762]: file "../../../../../../../../src/layer3/usr.sbin/rpd/lib/krt/async/krt_cnh_async.c", line 1293: "0"

Sep 26 09:27:25  hostname rpd[14298]: JTASK_ASSERT: Assertion failed rpd[14298]: file "../../../../../../../../src/layer3/usr.sbin/rpd/lib/krt/async/krt_cnh_async.c", line 1293: "0"

Sep 26 09:28:55  hostname rpd[14302]: JTASK_ASSERT: Assertion failed rpd[14302]: file "../../../../../../../../src/layer3/usr.sbin/rpd/lib/krt/async/krt_cnh_async.c", line 1293: "0"

Sep 26 09:30:44  hostname rpd[15110]: JTASK_ASSERT: Assertion failed rpd[15110]: file "../../../../../../../../src/layer3/usr.sbin/rpd/lib/krt/async/krt_cnh_async.c", line 1293: "0"

Sep 26 09:32:15  hostname rpd[15945]: JTASK_ASSERT: Assertion failed rpd[15945]: file "../../../../../../../../src/layer3/usr.sbin/rpd/lib/krt/async/krt_cnh_async.c", line 1293: "0"

Sep 26 09:33:48  hostname rpd[16822]: JTASK_ASSERT: Assertion failed rpd[16822]: file "../../../../../../../../src/layer3/usr.sbin/rpd/lib/krt/async/krt_cnh_async.c", line 1293: "0"

Sep 26 09:35:23  hostname rpd[17770]: JTASK_ASSERT: Assertion failed rpd[17770]: file "../../../../../../../../src/layer3/usr.sbin/rpd/lib/krt/async/krt_cnh_async.c", line 1293: "0"

Protocol sessions flap:

Sep 26 09:26:25  hostname  rpd[14298]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 169.254.6.198 (External AS 36351) changed state from OpenConfirm to Established (event RecvKeepAlive) (instance master)

Sep 26 09:26:29  hostname rpd[14298]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 169.254.6.202 (External AS 36351) changed state from OpenConfirm to Established (event RecvKeepAlive) (instance master)

Sep 26 09:26:35  hostname  rpd[14298]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 10.0.17.234 (External AS 36351) changed state from OpenConfirm to Established (event RecvKeepAlive) (instance master)

Continuous RPD cores:

root@hostname> show system core-dumps no-forwarding

-rw-rw----  1 root  wheel   65046308 Sep 26 09:28 /var/tmp/rpd.core-tarball.1.tgz

-rw-rw----  1 root  wheel   67385452 Sep 26 09:30 /var/tmp/rpd.core-tarball.2.tgz

-rw-rw----  1 root  wheel   82580621 Sep 26 09:32 /var/tmp/rpd.core-tarball.3.tgz

-rw-rw----  1 root  wheel   56510853 Sep 26 09:47 /var/tmp/rpd.core-tarball.4.tgz

Root Cause:
The issue is because of a known PR1738820: On Junos and Junos Evolved platforms, an RPD crash will be observed when RPD tries to add composite next-hop with the same parameters as available in the kernel’s existing composite next-hop which is marked deleted but not deleted due to some reference. When RPD tries to add composite next-hop into kernel with the same parameters, memory increases ,RPD crashes then generating continuous core, assertion failure and protocol flaps. This issue happens due to inconsistency between RPD and kernel for composite next-hop information.

Recovery Workaround until upgrading to fixed version:

1. Manual reboot of rpd [using “restart routing” CLI operational command]
2. If RPD restart did not help, try either of the following steps,    

          i. Reboot the switch 
         Ii. deactivate/activate all routing instances ,interfaces and Vlans by doing the below configuration changes:

               deactivate routing-instances
               deactivate interfaces
               deactivate vlans
               commit confirmed 2

Fixed version: 21.4R3-S5

PR Link: https://prsearch.juniper.net/problemreport/PR1738820