Description

Routing Engine[RE] crashes and VMcore is generated, which impacts traffic, when over a period of time MPLS next-hop is created/deleted frequently and the system is up for a very long time. The MPLS next-hop tag deletion counters end up having a value more than MPLS next-hop tag insertion counters.

Symptoms

On all Junos platforms, when the system is up for a very long time, tag stats counters increase and cross the 32-bit capacity when MPLS next-hop is created/deleted frequently. When this happens, the RE crashes.
If there is constant route churns resulting in frequest MPLS next-hop creation/deletion, it could also lead to RE crash.

This issue might be seen if the following conditions are met:
* On all Junos platforms
* MPLS next-hop creation and deletion frequently

A snippet of the logs taken during RE crash for reference:
Feb 19 19:29:22 lab_mx960 eventd[18410]: %SYSLOG-5-SYSTEM_ABNORMAL_SHUTDOWN: System abnormally shut down
Feb 19 19:29:22 lab_mx960 eventd[18410]: %SYSLOG-5-SYSTEM_OPERATIONAL: System is operational
Feb 19 19:29:23 lab_mx960 kernel: %KERN-2: panic: Assertion tag_control_stats.tag_iff_record_insertions >= tag_control_stats.tag_iff_record_deletions (file ../../../../../../../../../src/junos/bsd/sys/nettag/tag.c, line 763) failed!
Feb 19 19:29:23 lab_mx960 kernel: %KERN-2: cpuid = 3
Feb 19 19:29:23 lab_mx960 kernel: %KERN-3: jnx_update_system_state1: Called with panic str set to 0xffffffff80a14aa0, skipping..
Feb 19 19:29:23 lab_mx960 kernel: %KERN-2: Uptime: 165d8h40m20s


labroot@mx960> show chassis routing-engine | no-more
Routing Engine status:
  Slot 0:
    Current state                  Master
    Election priority              Master (default)
    Temperature                 28 degrees C / 82 degrees F
    CPU temperature             44 degrees C / 111 degrees F
    DRAM                      98254 MB (98304 MB installed)
    Memory utilization           5 percent
    15 min CPU utilization:
      User                      12 percent
      Background                 0 percent
      Kernel                    16 percent
      Interrupt                  6 percent
      Idle                      66 percent
    Model                          RE-S-2X00x6
    Serial ID                      EAAX0887
    Start time                     2024-02-19 19:27:43 -06
    Uptime                         7 hours, 37 minutes, 40 seconds
    Last reboot reason             0x1000:reboot due to panic
    Load averages:                 1 minute   5 minute  15 minute
                                       1.60       1.96       1.96

labroot@mx960> show system boot-messages 
Unlist request: unilist(nh index = 1123169) found on the rnhlist_deleted_root patnode, hence returning
Unlist request: unilist(nh index = 1162983) found on the rnhlist_deleted_root patnode, hence returning
panic: Assertion tag_control_stats.tag_iff_record_insertions >= tag_control_stats.tag_iff_record_deletions (file ../../../../../../../../../src/junos/bsd/sys/nettag/tag.c, line 763) failed!
cpuid = 3

Solution

The issue is due to PR1681892 and will be fixed in later Junos releases. Check out the below PR link for info on fixed Junos releases:
https://prsearch.juniper.net/problemreport/PR1681892  

As a workaround, following can also be executed proactively:
* If the system is a dual Routing Engine system, you can reboot the backup Routing Engine, do a GRES switchover to the backup Routing Engine, and reboot the old primary Routing Engine.
* If it is a single Routing Engine system, you can schedule a maintenance window and reboot it proactively.

 

Modification History

2024-02-27 : Article Created