This article explains about the multiple (more than 60) RDP core dumps generated when customer activate reconfigured Services, and when the VRF were activated that trigger it followed by multiple RE switchovers, and NSR out of sync issues.
Below are the core-dumps seen when activating all VRF's together when the below 3 conditions are met:
labroot@Router_1> show system core-dumps routing-engine both
re0:
--------------------------------------------------------------------------
/var/crash\/*core*: No such file or directory
-rw-rw---- 1 root wheel 372647114 Jun 25 01:12 /var/tmp/rpd.core-tarball.0.tgz
-rw-rw---- 1 root wheel 71848359 Jun 25 01:14 /var/tmp/rpd.core-tarball.1.tgz
-rw-rw---- 1 root wheel 62471735 Jun 25 01:15 /var/tmp/rpd.core-tarball.2.tgz
-rw-rw---- 1 root wheel 64486991 Jun 25 01:17 /var/tmp/rpd.core-tarball.3.tgz
-rw-rw---- 1 root wheel 50843542 Jun 25 05:00 /var/tmp/rpd.core-tarball.4.tgz
/var/tmp/pics\/*core*: No such file or directory
/var/crash/kernel.*: No such file or directory
/var/jails/rest-api/tmp\/*core*: No such file or directory
/tftpboot/corefiles\/*core*: No such file or directory
total files: 5
re1:
-rw-rw---- 1 root wheel 396363727 Jun 25 01:13 /var/tmp/rpd.core-tarball.0.tgz
-rw-rw---- 1 root wheel 69839917 Jun 25 01:14 /var/tmp/rpd.core-tarball.1.tgz
-rw-rw---- 1 root wheel 66336136 Jun 25 01:16 /var/tmp/rpd.core-tarball.2.tgz
-rw-rw---- 1 root wheel 59163582 Jun 25 01:17 /var/tmp/rpd.core-tarball.3.tgz
-rw-rw---- 1 root wheel 50132806 Jun 25 05:00 /var/tmp/rpd.core-tarball.4.tgz
-rw------- 1 root wheel 45758081 Jun 25 01:30 /var/tmp/rpd.core.0.gz
-rw------- 1 root wheel 50836852 Jun 25 01:23 /var/tmp/rpd.core.1.gz
{master}
labroot@Router_1> show log messages.9.gz | match "rpd.core"
Jun 25 01:11:26.036 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 01:12:54.287 2026 rdnvej300exh0630 dumpd[84039]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.0.tgz
Jun 25 01:13:24.344 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.1.gz
Jun 25 01:14:30.344 2026 rdnvej300exh0630 dumpd[85595]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.1.tgz
Jun 25 01:14:53.897 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 01:15:58.374 2026 rdnvej300exh0630 dumpd[89418]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.2.tgz
Jun 25 01:16:47.476 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 01:17:52.033 2026 rdnvej300exh0630 dumpd[91806]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.3.tgz
Jun 25 01:18:10.526 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.1.gz
Jun 25 01:19:14.709 2026 rdnvej300exh0630 dumpd[95405]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:20:01.649 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 01:21:06.189 2026 rdnvej300exh0630 dumpd[97970]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:21:23.968 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.1.gz
Jun 25 01:22:28.096 2026 rdnvej300exh0630 dumpd[1447]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:22:34.331 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 01:23:36.601 2026 rdnvej300exh0630 dumpd[3190]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:23:43.081 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.1.gz
Jun 25 01:24:42.096 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 01:24:46.908 2026 rdnvej300exh0630 dumpd[5179]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:25:41.895 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.1.gz
Jun 25 01:25:45.407 2026 rdnvej300exh0630 dumpd[6744]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:26:45.187 2026 rdnvej300exh0630 dumpd[9241]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:26:47.800 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 01:27:51.297 2026 rdnvej300exh0630 dumpd[11661]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:28:15.592 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 01:29:17.954 2026 rdnvej300exh0630 dumpd[13487]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:29:27.155 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.1.gz
Jun 25 01:30:28.933 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 01:30:30.816 2026 rdnvej300exh0630 dumpd[16337]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:31:32.057 2026 rdnvej300exh0630 dumpd[17822]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:49:25.751 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 01:50:28.252 2026 rdnvej300exh0630 dumpd[25735]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:50:31.628 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.1.gz
Jun 25 01:52:00.395 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 01:53:03.903 2026 rdnvej300exh0630 dumpd[31365]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:53:09.403 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.1.gz
Jun 25 01:54:14.463 2026 rdnvej300exh0630 dumpd[34276]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:54:37.711 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 01:55:41.293 2026 rdnvej300exh0630 dumpd[37052]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:55:46.187 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.1.gz
Jun 25 01:56:50.916 2026 rdnvej300exh0630 dumpd[39780]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:57:07.962 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 01:58:11.029 2026 rdnvej300exh0630 dumpd[42497]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:58:15.333 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.1.gz
Jun 25 01:59:19.803 2026 rdnvej300exh0630 dumpd[45445]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:59:36.595 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 02:00:39.336 2026 rdnvej300exh0630 dumpd[48117]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 02:00:43.667 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.1.gz
Jun 25 02:01:47.518 2026 rdnvej300exh0630 dumpd[50746]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 04:51:44.172 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 04:52:44.049 2026 rdnvej300exh0630 dumpd[17265]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 04:53:11.637 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 04:54:14.960 2026 rdnvej300exh0630 dumpd[20124]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 04:54:18.550 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.1.gz
Jun 25 04:55:10.519 2026 rdnvej300exh0630 dumpd[22458]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 04:56:03.856 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 04:57:08.344 2026 rdnvej300exh0630 dumpd[26153]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 04:57:13.728 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.1.gz
Jun 25 04:58:09.349 2026 rdnvej300exh0630 dumpd[29250]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 04:58:30.331 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 04:59:33.075 2026 rdnvej300exh0630 dumpd[32225]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 04:59:34.413 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.1.gz
Jun 25 05:00:27.567 2026 rdnvej300exh0630 dumpd[34755]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
{backup}
Jun 25 01:11:39.165 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 01:13:13.370 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.1.gz
Jun 25 01:14:16.814 2026 rdnvej300exh0630 dumpd[9644]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.1.tgz
Jun 25 01:15:10.877 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 01:16:17.683 2026 rdnvej300exh0630 dumpd[12203]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.2.tgz
Jun 25 01:16:41.880 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.1.gz
Jun 25 01:18:33.358 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 01:19:37.540 2026 rdnvej300exh0630 dumpd[18207]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:20:02.688 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.1.gz
Jun 25 01:21:58.794 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 01:23:02.656 2026 rdnvej300exh0630 dumpd[26457]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:26:37.327 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 01:27:40.009 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.2.gz
Jun 25 01:27:40.509 2026 rdnvej300exh0630 dumpd[12901]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:28:44.302 2026 rdnvej300exh0630 dumpd[15427]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:28:46.272 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.0.gz
Jun 25 01:29:47.516 2026 rdnvej300exh0630 dumpd[17717]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:49:27.613 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.2.gz
Jun 25 01:50:30.267 2026 rdnvej300exh0630 dumpd[19217]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:50:54.711 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.3.gz
Jun 25 01:51:58.475 2026 rdnvej300exh0630 dumpd[22211]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:52:00.530 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.2.gz
Jun 25 01:53:04.256 2026 rdnvej300exh0630 dumpd[25305]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:53:27.588 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.3.gz
Jun 25 01:54:31.179 2026 rdnvej300exh0630 dumpd[28230]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:54:32.901 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.2.gz
Jun 25 01:55:37.094 2026 rdnvej300exh0630 dumpd[31277]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:55:55.936 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.3.gz
Jun 25 01:56:58.625 2026 rdnvej300exh0630 dumpd[34041]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:57:02.302 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.2.gz
Jun 25 01:58:06.075 2026 rdnvej300exh0630 dumpd[37140]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:58:25.381 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.3.gz
Jun 25 01:59:28.649 2026 rdnvej300exh0630 dumpd[39898]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 01:59:32.084 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.2.gz
Jun 25 02:00:36.169 2026 rdnvej300exh0630 dumpd[42891]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 02:00:52.823 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.3.gz
Jun 25 02:01:55.406 2026 rdnvej300exh0630 dumpd[44205]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 04:51:44.987 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.2.gz
Jun 25 04:52:48.210 2026 rdnvej300exh0630 dumpd[15730]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 04:52:51.070 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.3.gz
Jun 25 04:53:40.264 2026 rdnvej300exh0630 dumpd[18046]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 04:54:36.742 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.2.gz
Jun 25 04:55:40.513 2026 rdnvej300exh0630 dumpd[21755]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 04:55:44.825 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.3.gz
Jun 25 04:56:35.119 2026 rdnvej300exh0630 dumpd[24111]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 04:57:14.250 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.2.gz
Jun 25 04:58:17.604 2026 rdnvej300exh0630 dumpd[27741]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
Jun 25 04:58:21.593 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.3.gz
Jun 25 04:59:48.104 2026 rdnvej300exh0630 logger: JLAUNCHD_PROC_COREDUMPED: core_file=/var/tmp/rpd.core.2.gz
Jun 25 05:00:51.375 2026 rdnvej300exh0630 dumpd[32288]: Core and context for rpd saved in /var/tmp/rpd.core-tarball.4.tgz
(gdb) bt
#0 thr_kill () at thr_kill.S:4
#1 0x000000003d3bebc1 in __raise (s=s@entry=6) at /volume/build/junos/occam/llvm-13.0/sandbox-bsd15-244-20250701-5b0ec39/freebsd/bsd15_244/20250701.220450_builder_bsd15_244.5b0ec39/src/lib/libc/gen/raise.c:48
#2 0x000000003d45c849 in abort () at /volume/build/junos/occam/llvm-13.0/sandbox-bsd15-244-20250701-5b0ec39/freebsd/bsd15_244/20250701.220450_builder_bsd15_244.5b0ec39/src/lib/libc/stdlib/abort.c:61
#3 0x0000000002bdef0a in __task_quit__ (code=) at ../../../../../../src/daemon-infra/lib/libjtask/base/task_quit.c:127
#4 0x0000000002ba5c56 in task_quit (code=code@entry=22) at ../../../../../../src/daemon-infra/lib/libjtask-jthr/jtask_jthr_quit.c:53
#5 0x0000000002bf5f4f in task_assert (file=0x9f70b1 "../../../../../../../../src/layer3/usr.sbin/rpd/lib/rt/rt_directrt_frr.c", line=line@entry=4218, test=0xa8f9f7 "frr_nh") at ../../../../../../src/daemon-infra/lib/libjtask/base/task_assert.c:89
#6 0x000000000278ff09 in rt_directrt_form_list_nh (rt=, rt@entry=0x2ea2fe767808, rt_entry_protect=rt_entry_protect@entry=0x2ea313d8b798) at ../../../../../../../../src/layer3/usr.sbin/rpd/lib/rt/rt_directrt_frr.c:4218
#7 0x000000000278a682 in rt_directrt_hfrr_rt_add (hfrrp=hfrrp@entry=0x2ea2c761f800, gwh_arp=0x2ea2d6bb6100, gwh_frr=gwh_frr@entry=0x2ea2d6bb6180, rtt=rtt@entry=0x2ea2c92bfa00, rt=rt@entry=0x2ea2fe767808, rt_entry_protect=rt_entry_protect@entry=0x2ea313d8b798) at ../../../../../../../../src/layer3/usr.sbin/rpd/lib/rt/rt_directrt_frr.c:4263
#8 0x000000000278a321 in rt_directrt_arp_rt_add_msg_process (hfrrp=hfrrp@entry=0x2ea2c761f800, rtparms=...) at ../../../../../../../../src/layer3/usr.sbin/rpd/lib/rt/rt_directrt_frr.c:3850
#9 0x00000000026255b9 in krt_directrt_add_arp_rt (tp=tp@entry=0x2ea2e1ea6800, hfrrp=0x18a3b, hfrrp@entry=0x2ea2c761f800, dest=dest@entry=0x2ea30ce9e250, plen=plen@entry=32, rtm=, rtth=rtth@entry=0x2ea2c92bfa00) at ../../../../../../../../src/layer3/usr.sbin/rpd/lib/krt/krt_directrt_frr.c:384
#10 0x000000000278c0a6 in rt_directrt_process_route_add_cb (tp=0x2ea2e1ea6800, iflname=0x2ea2f8549de0 "xe-7/4/2:0.105", rtih=, dest=0x2ea30ce9e250, plen=32, rtth=0x2ea2c92bfa00, rtm=0x2ea2d243f008) at ../../../../../../../../src/layer3/usr.sbin/rpd/lib/rt/rt_directrt_frr.c:5297
#11 0x0000000002678333 in krt_route_add_cb_walk (tp=tp@entry=0x2ea2e1ea6800, iflname=0x2ea2f8549de0 "xe-7/4/2:0.105", rtih=rtih@entry=0x2ea30f8a1000, dest=dest@entry=0x2ea30ce9e250, plen=plen@entry=32, rtth=0x2ea2c92bfa00, rtm=0x2ea2d243f008) at ../../../../../../../../src/layer3/usr.sbin/rpd/lib/krt/krt_read_route.c:1123
#12 0x0000000002673905 in krt_decode_directrt (tp=tp@entry=0x2ea2e1ea6800, msg=msg@entry=0x2ea2d243f008) at ../../../../../../../../src/layer3/usr.sbin/rpd/lib/krt/krt_decode_rt.c:1832
#13 0x0000000002677b64 in krt_create_and_send (af=2, ifl_index=..., krt_sync_data_ptr=, tp=, krtp=, rt_table_idx=) at ../../../../../../../../src/layer3/usr.sbin/rpd/lib/krt/krt_read_route.c:787
#14 krt_read_from_kernel (jp=) at ../../../../../../../../src/layer3/usr.sbin/rpd/lib/krt/krt_read_route.c:883
#15 0x0000000002bc57ec in task_job_run_common (jp=jp@entry=0x2ea2c7e227e0, jctx=jctx@entry=0x2ea2c7f11c80, dispatcher=0x87ec95 "background dispatch") at ../../../../../../src/daemon-infra/lib/libjtask/base/task_jobs.c:1000
#16 0x0000000002bc66c6 in task_job_run_job_bg (scheduler=0x3091300 , jobs_var=0x3091090 , jp=0x2ea2c7e227e0, log_msg=) at ../../../../../../src/daemon-infra/lib/libjtask/base/task_jobs.c:1220
#17 task_job_bg_dispatch () at ../../../../../../src/daemon-infra/lib/libjtask/base/task_jobs.c:1436
#18 0x0000000002c02f0a in task_scheduler_internal (enable_slip_detector=true, no_exit=false) at ../../../../../../src/daemon-infra/lib/libjtask/base/task_scheduler.c:314
#19 0x0000000002c02a3e in task_scheduler () at ../../../../../../src/daemon-infra/lib/libjtask/base/task_scheduler.c:584
#20 0x0000000001189ed6 in rpd_main (argc=, argv=) at ../../../../../../../src/layer3/usr.sbin/rpd/os/rpd_main.c:675
#21 main (argc=, argv=) at ../../../../../../../src/layer3/usr.sbin/rpd/os/rpd_main.c:444
The environment under which this occurs is also clear and requires the following three conditions:
Expanding on point 3, if the preferred backup route is via the Core (from a remote PE router), the issue will not occur, even if the route is still advertised by the local eBGP peer. See the detailed output below for some CLI reference to clarify.
Even if the core routes are more preferred than the eBGP routes, we are still at risk in the event of an RPD restart or device reboot. In this case, the PE may learn routes from the local eBGP path before receiving them from the core, leading to all 3 conditions being met. This is why the RPD crash was cyclical in production.
CLI captures (detailed reference only)
To provide the trigger I have simulated an eBGP peer to advertise the VLAN 105 subnet 192.148.130.32/27 on the lab PE router. This is also advertised from a remote lab PE router Initially the preferred backup is via the Core and Host FRR is NOT enabled at this stage:
In this state, Condition 3 is not met, so if HFRR is enabled, the routes can be instantiated and the device is stable:
{master}[edit]
remote1@rdnvej300exh0630# activate routing-instances S1000376R routing-options interface et-7/4/0.105
remote1@rdnvej300exh0630# commit
configuration check succeeds
commit complete
192.148.130.34/32 @[ARP/4294967293] 00:00:18, from 192.148.130.32
Unusable
#[FRR/200] 00:00:18, from 192.148.130.32
> to 192.148.130.34 via et-7/4/0.105
to 172.18.56.12 via ae0.0, Push 5261, Push 604617(top)
to 172.18.56.14 via ae1.0, Push 5261, Push 604617(top). <====== Backup path via the Core, wipm-e-305
192.148.130.35/32 *[Local/0] 10:27:47
Local via et-7/4/0.105
192.148.130.37/32 @[ARP/4294967293] 00:00:18, from 192.148.130.32
> to 192.148.130.37 via et-7/4/0.105
to 172.18.56.14 via ae1.0, Push 5261, Push 604617(top)
If we block the routes coming the core network, meaning only the IBR/eBGP route can be used as a backup then RPD will crash. We can also trigger this by restarting RPD.
##### RPD Crashes, leading to a switchover ….etc.
remote1>
Message from syslogd@ at Jun 29 21:25:05 ...
fpc7 CMLC: Going disconnected; Routing engine chassis socket closed abruptly
fpc0 CMLC: Going disconnected; Routing engine chassis socket closed abruptly
Message from syslogd@ at Jun 29 21:25:08 ...
last message repeated 2 times
Engaged Engineering and after Dev reviewed the backtrace, trigger and logs
gdb)
4205 rt_template_initialize_and_grow_ngw(&lenht, 2);
4206 rt_template_set_rtn_type(lenht, RT_NH_LIST);
4207 rt_template_set_rtn_subtype(lenht, RT_NH_SUBTYPE_HFRR_UNL);
4208 rt_template_set_rtn_ngw(lenht, 2);
4209 rtg_0 = rt_template_get_gateway(0, lenht);
4210 rtg_1 = rt_template_get_gateway(1, lenht);
4211
4212 memcpy(rtg_0, rt_nexthops_gw(arp_nh, 0), sizeof(rt_gateway));
4213 memcpy(rtg_1, rt_nexthops_gw(frr_backup_nh, 0), sizeof(rt_gateway));
4214
(gdb)
4215 rtg_0->rtg_weight = RTG_IGP_FRR_WEIGHT;
4216 rtg_1->rtg_weight = RTG_IGP_BACKUP_WEIGHT;
4217
4218 assert(frr_nh); >>>>>> frr_nh variable is started with NULL at the start of the function
4219 rt_nexthops_set_subtype(frr_nh, RT_NH_SUBTYPE_HFRR_UNL);
4220 frr_nh = rt_nexthops_find(lenht);
4221
4222 /* For old style unilist NH, the components NHs need not be in NH tree. */
4223 rt_nh_free(frr_backup_nh);
4224 rt_nh_free(arp_nh);
the frr_nh var in the line number 4218 above is NULL as its initialized at the function start with NULL.
If we see the code closely here, we are doing an assert first on the var and then assigning a value to it.
Ideally, we should do the reverse, meaning doing rt_nexthops_find() API call and then do assert on the return value.
The above code is a buggy code, and is always bound to crash if the control reaches this piece of code.
As per which the root cause is as below:
Assert check before memory allocation which is a Day-1 issue.
Please upgrade to the below fixed junos release:
Resolved-In
evo:22.3X50-EVO evo:22.3X50-J5-EVO evo:22.3X80-D49-EVO evo:25.4R1-S3-EVO evo:26.2X200-D10-EVO junos:21.2R3-S11 junos:22.3R3-S5 junos:22.4R3-S9 junos:23.4R2-S8 junos:24.4R2-J17 junos:25.4R2 junos:26.3R1
Tested the fixed junos version in lab and the device is no more trigering the RPD core-dump even if all the above 3 conditions are met.