Routing Engine[RE] crashes and VMcore is generated, which impacts traffic, when over a period of time MPLS next-hop is created/deleted frequently and the system is up for a very long time. The MPLS next-hop tag deletion counters end up having a value more than MPLS next-hop tag insertion counters.
On all Junos platforms, when the system is up for a very long time, tag stats counters increase and cross the 32-bit capacity when MPLS next-hop is created/deleted frequently. When this happens, the RE crashes. If there is constant route churns resulting in frequest MPLS next-hop creation/deletion, it could also lead to RE crash. This issue might be seen if the following conditions are met: * On all Junos platforms * MPLS next-hop creation and deletion frequently
A snippet of the logs taken during RE crash for reference:Feb 19 19:29:22 lab_mx960 eventd[18410]: %SYSLOG-5-SYSTEM_ABNORMAL_SHUTDOWN: System abnormally shut downFeb 19 19:29:22 lab_mx960 eventd[18410]: %SYSLOG-5-SYSTEM_OPERATIONAL: System is operationalFeb 19 19:29:23 lab_mx960 kernel: %KERN-2: panic: Assertion tag_control_stats.tag_iff_record_insertions >= tag_control_stats.tag_iff_record_deletions (file ../../../../../../../../../src/junos/bsd/sys/nettag/tag.c, line 763) failed!Feb 19 19:29:23 lab_mx960 kernel: %KERN-2: cpuid = 3Feb 19 19:29:23 lab_mx960 kernel: %KERN-3: jnx_update_system_state1: Called with panic str set to 0xffffffff80a14aa0, skipping..Feb 19 19:29:23 lab_mx960 kernel: %KERN-2: Uptime: 165d8h40m20slabroot@mx960> show chassis routing-engine | no-more Routing Engine status: Slot 0: Current state Master Election priority Master (default) Temperature 28 degrees C / 82 degrees F CPU temperature 44 degrees C / 111 degrees F DRAM 98254 MB (98304 MB installed) Memory utilization 5 percent 15 min CPU utilization: User 12 percent Background 0 percent Kernel 16 percent Interrupt 6 percent Idle 66 percent Model RE-S-2X00x6 Serial ID EAAX0887 Start time 2024-02-19 19:27:43 -06 Uptime 7 hours, 37 minutes, 40 seconds Last reboot reason 0x1000:reboot due to panic Load averages: 1 minute 5 minute 15 minute 1.60 1.96 1.96labroot@mx960> show system boot-messages Unlist request: unilist(nh index = 1123169) found on the rnhlist_deleted_root patnode, hence returningUnlist request: unilist(nh index = 1162983) found on the rnhlist_deleted_root patnode, hence returningpanic: Assertion tag_control_stats.tag_iff_record_insertions >= tag_control_stats.tag_iff_record_deletions (file ../../../../../../../../../src/junos/bsd/sys/nettag/tag.c, line 763) failed!cpuid = 3
The issue is due to PR1681892 and will be fixed in later Junos releases. Check out the below PR link for info on fixed Junos releases:https://prsearch.juniper.net/problemreport/PR1681892 As a workaround, following can also be executed proactively:* If the system is a dual Routing Engine system, you can reboot the backup Routing Engine, do a GRES switchover to the backup Routing Engine, and reboot the old primary Routing Engine.* If it is a single Routing Engine system, you can schedule a maintenance window and reboot it proactively.