Traffic impact might be seen on SPC3 card as TX/RX queue getting stuck

PR1671649 - Platforms with SPC3 card, Flowd restart or PIC (Physical Interface Card) going offline/online may cause SPC3's sending of packets to get stuck.

In case of flowd crash, flowd would be restarted and the PICs would go through offline/online simultaneously. Whenever a PIC goes through offline/online in sequence and in the meantime the ingress traffic keeps coming in, the TX-stuck issue could be triggered. Also a user can manually set PIC offline/online with the pic-online/offline commands and trigger the same issue.
 

# show i386 fpga dma-counter   
RX DMA counter:
PCI 0: 1682641
PCI 1: 2151597
PCI 2: 2255388
PCI 3: 1973098
Total: 8062724
TX DMA counter:
PCI 0: 1687220
PCI 1: 2157281
PCI 2: 2260779
PCI 3: 0  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Total: 6105280

Command from flowd "show i386 fpga datapath-stats brief" can show "tx_NoDp_drop" counter incrementing when this issue has occurred. Below logs could be observed:

fpc7 daemon.err srxpfe[8484]: Not enough free TxDesc, pci2, tq4, t_tail 1517 , ccnt = 693000000!!!
fpc7 daemon.err srxpfe[8484]: Not enough free TxDesc, pci2, tq8, t_tail 1181 , ccnt = 694000000!!!
fpc7 daemon.err srxpfe[8484]: Not enough free TxDesc, pci3, tq20, t_tail 550 , ccnt = 695000000!!!

PR1664517 - This issue happens when FPGA in the hardware inserts the Rx packets into the wrong Rx descriptor entries. This causes direct memory access to stuck which causes an impact on the processing of further packets.

Command from flowd "show i386 fpga dma-counter" would show the counter not being incremented in the problem state.

# show i386 fpga dma-counter   
RX DMA counter:
PCI 0: 694102
PCI 1: 0  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
PCI 2: 720746
PCI 3: 737030
Total: 2151878
TX DMA counter:
PCI 0: 693583
PCI 1: 78
PCI 2: 720109
PCI 3: 736270
Total: 2150040

Below outputs needs to be collected during the time of the issue.
 

# show talus version
# show i386 fpga dma-counter <<<<< 2-3 Times
# show i386 fpga datapath-stats brief <<<< 2-3 times

This would help us to check if the Rx and TX DMA counter are stuck or incrementing.

• In the event of TX queue stuck or not incrementing, we might be encountering PR1671649
  • To recover from the issue, Reboot SPC3 card.
• In the event of RX queue stuck or not incrementing, we might be encountering - PR1664517
  • To recover from the issue, the PIC needs to be restarted.

Note: In order to fix the issue or prevent it from happening again, the device needs to be upgraded to a fixed Junos version.

2023-07-31: Initial publication