Description

This article addresses common issues with network devices, including:


  1. Device not responding to SNMP requests.
  2. CLI slow to show certain commands.
  3. Issues worsening after switching over between Routing Engines (REs).


Symptoms

Device not responding to SNMP requests: Network monitoring tools fail to retrieve data from the device.

CLI slow to show certain commands: Command Line Interface (CLI) takes an unusually long time to execute commands.

Switching over between REs worsens symptoms: Attempting to switch between primary and backup Routing Engines exacerbates the issues.



{master}

MX_router> show interfaces ae0 | match rate

Input rate : 0 bps (0 pps)

Output rate : 0 bps (0 pps)


{master}

MX_router> show isis adjacency | match ae0

ae0.0 TEST_DEVICE 2 Up 20



{master}

MX_router> show interfaces ae0 | match rate

Input rate : 7990260120 bps (1417440 pps)

Output rate : 8556641296 bps (1536211 pps)



Customer might not see unusual CPU/memory usage, or anything in log messages or chassisd that makes suspicious. JTAC identified an issue with communication between PFE and RE’s and thinks that our HA (GRES/NSR) configuration may be making the issue persistent across RE toggles – as the backup builds a copy of bad data from the primary, it then uses that when it takes mastership.

Solution

JTAC Recommended Steps

  1. Disable GRES/NSR:
deactivate chassis redundancy graceful-switchover
deactivate routing-options nonstop-routing
  1. Reload Backup RE (re1):
    • Reload the backup Routing Engine (re1).
    • Confirm that re1 comes up properly before toggling over to it.
    • Note: This will cause a downtime of 10-15 minutes as all the linecards reestablish.
    • Check if interfaces (e.g., ae0) are reporting packets in/out.
  1. Switch Over to re1:
    • Once re1 is up and running, switch over to it.
  1. Reload Primary RE (re0):
    • Reload the primary Routing Engine (re0).
  1. Re-enable HA:
activate chassis redundancy graceful-switchover
activate routing-options nonstop-routing
  1. Wait for Redundancy to Reestablish:
    • Allow time for the redundancy to reestablish.
  1. Fail Back Over to re0:
    • Fail back over to re0 to normalize the setup. This should be more graceful.

Following these steps should help purge the bad data and stabilize your system. If you encounter any issues during the process, it's a good idea to keep JTAC informed for further assistance.

Modification History

2025-02-18 : Article Created