This article details an issue affecting QFX5200 switches, where large-scale next-hop changes can trigger an FPC crash, especially when a PFE table exceeds 80% capacity.

FPC crash and reboots and a DCPFE core dump is generated.

root@qfx5200> show chassis fpc detail
Slot 0 information:
 State                Online
 Temperature          41 degrees C / 105 degrees F
 Total CPU DRAM       16384 MB
 Total SRAM           0 MB
 Total SDRAM          0 MB
 Start time           2025-01-09 11:43:30 PST
 Uptime               37 seconds

root@qfx5200> show system core-dumps
localre:
--------------------------------------------------------------------------
/var/tmp/*core*: No such file or directory
Host Crash Info:
--------------------------------------------------------------------------
-rw-r--r-- 1 root root  15870617 Jan 9 11:43 /var/crash/core-dcpfe-dcpfe0-1-1736451794.core.tgz

The topology walk triggered by the PFE to reinstall routes during next-hop changes can generate substantial CPU overhead during large-scale updates, potentially leading to a watchdog crash. This can be further exacerbated by hash collisions, which become more likely when the host table exceeds 80% capacity with LPM profile. Hash collision resolution is also a CPU-intensive process.

A DCPFE crash will cause a complete outage but will recover automatically. Network convergence events and ECMP or LAG changes are among the causes of large-scale next-hop changes. The “show pfe route summary hw” command can be used to monitor PFE table utilization.

This issue has been addressed via PR 1854995. Refer to the PR or contact JTAC for a list of releases that contain the fix.

The fix consists of two components:

Efficient Handling of Next-Hop Changes:

The new scheme updates next-hop entries directly in hardware, eliminating the need to transition each route individually from the old next-hop to the new one. This approach significantly reduces CPU usage when handling a large number of routes.

Optimized Route Management to Prevent Hash Collision:

Hash collisions become more likely when the host route table exceeds 80% occupancy for LPM profile. To mitigate this, the new scheme bypasses the route_add operation in the host table beyond this threshold and directly adds routes to the LPM (Longest Prefix Match) table. This prevents unnecessary trial insertions into the host table, further reducing CPU overhead.

For a 30K re-ARP test, the results show up to a 4x improvement in CPU efficiency with the newer code, reducing processing overhead and improving system performance.

PFE table overview with the fix: 

root@switch> show pfe route summary hw
Unit: 0
Profile active: lpm-profile
Type            Max       Used      Free      % free
----------------------------------------------------
IPv4 Host       8192      6546      1638      20.00       <<<<< 20% is reserved
IPv4 LPM        196608    183481    13127     6.68         
IPv4 Mcast      4096      0         819       20.00
 
IPv6 Host       4096      4         819       20.00
IPv6 LPM(< 64)  32768     4         32764     99.99
IPv6 LPM(> 64)  32768     0         32768     100.00
IPv6 Mcast      2048      0         410       20.02