SSH stops working in a Standalone QFX5100-48S.

Logs like this can be found in message logs:

Jan 18 19:37:21 tuc-pod-00-leaf-05 /kernel: rts_veto_sys_check_low_on_kmem: Memory over consumed. Op 2 err 55, rtsm_id 0:-1, msg type 10, veto simulation: 0.

Jan 18 19:37:21 tuc-pod-00-leaf-05 /kernel: rts_veto_sys_check_low_on_kmem: free kmem_map memory = (14430208) curproc = rpd.

Jan 18 19:37:23 tuc-pod-00-leaf-05 /kernel: rts_veto_sys_check_low_on_kmem: Memory over consumed. Op 2 err 55, rtsm_id 26:-1, msg type 6, veto simulation: 0.

Jan 18 19:37:23 tuc-pod-00-leaf-05 /kernel: rts_veto_sys_check_low_on_kmem: free kmem_map memory = (14458880) curproc = l2ald.

Jan 18 19:37:26 tuc-pod-00-leaf-05 /kernel: rts_veto_sys_check_low_on_kmem: Memory over consumed. Op 1 err 55, rtsm_id 0:-1, msg type 2, veto simulation: 0.

Jan 18 19:37:26 tuc-pod-00-leaf-05 /kernel: rts_veto_sys_check_low_on_kmem: free kmem_map memory = (14458880) curproc = l2ald.

NSR is enabled:

routing-options {

  router-id 11.30.10.5;

  autonomous-system 399900;

  static {

    route 0.0.0.0/0 next-hop 10.2.0.1;

  }

  nonstop-routing;

  forwarding-table {

    export PFE-LB;

  }

}

But the device is a standalone:

user@QFX5100> show virtual-chassis status

Virtual Chassis ID: ae39.0183.6a61

Virtual Chassis Mode: Enabled

                        Mstr      Mixed Route Neighbor List

Member ID Status  Serial No  Model     prio Role   Mode Mode ID Interface

0 (FPC 0) Prsnt  XXXXXX qfx5100-48s-6q 128  Master*   N VC ---> No more members.

Member ID for next new member: 1 (FPC 1)

These logs are related to a kernel memory leak. In most cases the reported problems were related to NSR (nonstop-routing feature).

In this case, the switch has NSR enable, despite having only one routing engine (Not a Virtual chassis) which is not expected.

NSR/GRES is used in EX Series with multiple Routing Engines or on an EX-Series or a QFX series switch in a Virtual Chassis or Virtual Chassis Fabric configuration.

So, there would be kernel temp memory leak that would hit a threshold, once this threshold is hit, kernel will run out of sufficient memory and due to low memory, it may not process any more requests and stops allocating memory. One of the most commonly affected processes is SSH.

In similar cases, as a workaround, nonstop-routing was deactivated.

Since NSR is enabled in a standalone device, could be this is triggering an unwanted behavior. If possible, disable NSR and monitor.

Lastly, if the issue continues after disabling NSR, please reboot the device and monitor.