Description

On SRX & MX platforms, TX DMA counter would be stuck to value 0 and no packet can be transferred by the SPC3 card. 
It could happen only during the system/SPC3 bootup time. The "packet loss" can be seen when the issue happens.

Symptoms

This issue might be seen if the following conditions are met:
* On SRX and MX platforms
* SPC3 card 
* System/SPC3 bootup time

Check the output for command "show i386 fpga dma-counter" and see if PCI DMA counter is showing stuck in TX queue

FLOWD_SPC3-7.0(vty)# show i386 fpga dma-counter
TX DMA counter:
PCI 0: 1730111
PCI 1: 1871049
PCI 2: 1469660
PCI 3: 0          >>>>>>>> Counter stuck at 0 value
Total: 5070820

Solution

During the SPC3 card boot-up, in a rare scenario once the card comes up user may see Tx DMA counter stuck at 0 value.

It causes all packets getting stuck in TX buffer and none of the packets are transmitted out.

Actually Talus FPGA txdma_csr clear bit is a floating value. During SPC3 card initialisation if the clear bit happens to get set to a non-zero (higher) value which cause Tx queue to get stuck and no packet can be transferred by the SPC3 card

 

Data Collection to be done:

start shell user root
vty fpc7.0

show i386 fpga dma-counter
FLOWD_SPC3-X.0(vty)# show i386 fpga dma-counter
TX DMA counter:
PCI 0: 1730111
PCI 1: 1871049
PCI 2: 1469660
PCI 3: 0          >>>>>>>> Counter stuck at 0 value
Total: 5070820
 
bringup jspec read talus_pcie[0] register dma_txdesc txdma_csr     >>>>>>>>> Check for value of clear-bit, a non-zero value shows problematic state 
0x00057000  talus.dma_txdesc.txdma_csr         0000010E   
                                              cdone[9:9] : 0x0
                                              clear[8:8] : 0x1       >>>>>>>>>> Clear bit has non-zero value (0x1)
                                             dma_en[3:3] : 0x1
                                           dma_init[2:2] : 0x1
                                              pdone[1:1] : 0x1
                                                rst[0:0] : 0x0
show i386 fpga data-stats brief      >>>>>>>>> Check for Tx queue showing stuck 
FLOWD_SPC3-7.0(vty)# show i386 fpga datapath-stats brief
< -------- snipped ------------ >
----------- tx_good_pkt  tx_NoDp_drop tx_stall_dp  tx_mbuf_free rx_good_pkt  rx_desc_err  rx_dma_dv_0  rx_mbuf_allo q-dp pci-dp 
core21:0--- 2047         4648125      0            0            0            0            0            0            yes  yes 
core21:1--- 0            0            0            0            0            0            0            0            yes  yes 
core21:2--- 0            0            0            0            0            0            0            0            yes  yes 
core21:3--- 0            0            0            0            0            0            0            0            yes  yes 
----------- tx_good_pkt  tx_NoDp_drop tx_stall_dp  tx_mbuf_free rx_good_pkt  rx_desc_err  rx_dma_dv_0  rx_mbuf_allo q-dp pci-dp 
core22:0--- 2047         4693438      0            0            0            0            0            0            yes  yes 
core22:1--- 0            0            0            0            0            0            0            0            yes  yes 
core22:2--- 0            0            0            0            0            0            0            0            yes  yes 
core22:3--- 0            0            0            0            0            0            0            0            yes  yes
show i386 fpga descriptor 1 <descriptor number> [0-3]     >>>>>>>>> Check for dv-bit, a non-zero value shows problematic state
 fpc8.pic0> show i386 fpga descriptor 1 22 0
 
        tx descriptor      length    dest dv
     0  0x000000197d2de06c 0x0040  0x0028 1 0x000000197d2de06c 0x0028    <<<< dv bit is set 
     1  0x000000197d2dd86c 0x0040  0x0029 1 0x000000197d2dd86c 0x0029    <<<< dv bit is set
     2  0x000000197d2dd06c 0x0040  0x0029 1 0x000000197d2dd06c 0x0029    <<<< dv bit is set

 

A PIC/FPC reboot is needed for the SPC3 card to recover from the problematic state

 

Please check the following link for more details - https://prsearch.juniper.net/problemreport/PR1706756
 

Modification History

Details added

Related Information

https://supportportal.juniper.net/s/article/MX-Customer-is-seeing-partial-service-impact-on-SPC3-card-after-observing-one-of-the-PCIE-link-getting-stuck-in-TX-direction