The SRX4600 device is encountering substantial packet loss across multiple interfaces.
This issue appears to be linked to the FPGA processing of high-priority traffic, particularly affecting ARP resolution and other critical protocol messages.
High packet loss observed on multiple interfaces.
ARP requests are sent but responses do not reach the control plane. Ping failures to directly connected devices. Routing protocols such as BGP, OSPF, and ISIS may experience disruptions.
The issue is present on SRX4600 and running a FPGA v1.63 or later
Cause:
When a packet arrives at an SRX4600 interface, it is assembled into a fabric cell and sent to the Glacis FPGA.
The FPGA consists of 24 cores (excluding cores 13 and 14), each having four queues (Q0 to Q3).
This is due to the scenario where excessive high-priority traffic through Q1 results in CPU backpressure to the FPGA.
This backpressure overloads the FPGA, causing Q1 to freeze in specific cores, leading to packet drops and incomplete transmissions to the CPU.
Verification:
To check the FPGA version, use the following command:
user@SRX4600> start shell pfe network node0.fpc0
Then, check the FPGA version:
FLOWD_SRX4600(SRX4600- vty)# show i386 fpga version
Example output:
FPGA Version: 0x00000175 <<< FPGA version 1.75
To verify that the issue is there, monitor the datapath statistics and determine if rx_good_pkt is not incrementing:
During problematic state, in order to verify that the issue is there you can monitor the datapath statistics the rx_good_pkt counter on the high-priority queue (Q1) for some cores does not increment and you will see rx_desc_err incrementing
>start shell user root
% vty fpc0 (or vty nodex.fpc0 in case of cluster setup)
=== SPM0, node0.fpc0> show i386 fpga datapath-stats extensive
----------- tx_good_pkt tx_NoDp_drop tx_bufptr TxPkt_size_0 TxPkt<64B TxPkt>9KB tx_stall_dp tx_mbuf_free tx_func_call rx_good_pkt rx_desc_err
core15:1--- 0 0 0 0 0 0 0 0 0 1716049 1550499
core16:1--- 0 0 0 0 0 0 0 0 0 1519937 1724317
core17:1--- 0 0 0 0 0 0 0 0 0 1995201250 1752351
core15:1--- 0 0 0 0 0 0 0 0 0 1716049 1560241
core16:1--- 0 0 0 0 0 0 0 0 0 1519937 1734406
core17:1--- 0 0 0 0 0 0 0 0 0 1995201250 1762628
1--- 0 : 1 is the affected queue
rx_desc_err increments during problematic state
rx_good_pkt does not increment
Rebooting the affected node temporarily clears the stuck queue, restoring functionality.
Implementing a static ARP entry bypasses the ARP resolution process, providing a temporary solution. However, the root cause remains unaddressed, and Q1 could still become stuck again, affecting other protocols.
Permanent Fix: A permanent fix is being tracked under the following PR:
https://prsearch.juniper.net/problemreport/PR1823577