Description

This article explains DPDK port stuck issue reported on SRX4100/4200 Platforms and Junos versions where it is fixed.

Symptoms

To verify the nic state, please follow below steps:
  • Go to shell mode: “start shell”
  • Connect to host: “ssh -JU __juniper_private4__ [email protected]
  • Check the mapping of interface with Nic port: “show nic stats port <0-8>”

(Select the port at which you are seeing the issue)

  • This is the command to collect the statistics for the given Nic port: “show nic queue-stats port <0-8>”

(Collect this output multiple times to see if “TX-packets” counters are increasing continuously or not)

  • If you notice that “TX packets” counters are not incrementing then look into logs for this message “DPDK port # is stuck with hardware issue” (Note: 19.4R3 onwards only, this log message will be seen)

Example:

Lab-Firewall#show nic stats port 0" target node0.fpc0

================ tnp_0x10001081 ================

SENT: Ukern command: show nic stats port 0

 

 ######################## NIC statistics for port 0 ########################

 RX-packets: 488683     RX-errors: 0          RX-bytes: 313133530

 RX-nombuf: 0

 TX-packets: 73967      TX-errors: 0          TX-bytes: 22554358                       <<<<<

 Stats reg 0 RX-packets: 61717      RX-errors: 0          RX-bytes: 30222704

 Stats reg 1 RX-packets: 45396      RX-errors: 0          RX-bytes: 19900315

 Stats reg 2 RX-packets: 37026      RX-errors: 0          RX-bytes: 24100888

 Stats reg 3 RX-packets: 36384      RX-errors: 0          RX-bytes: 13191281

 Stats reg 4 RX-packets: 38980      RX-errors: 0          RX-bytes: 15080392

 Stats reg 5 RX-packets: 44501      RX-errors: 0          RX-bytes: 17099781

 Stats reg 6 RX-packets: 46914      RX-errors: 0          RX-bytes: 30682620

 Stats reg 7 RX-packets: 133412     RX-errors: 0          RX-bytes: 143082867

 Stats reg 8 RX-packets: 43447      RX-errors: 0          RX-bytes: 19697040

 Stats reg 9 RX-packets: 906        RX-errors: 0          RX-bytes: 75642

 Stats reg 10 RX-packets: 0          RX-errors: 0          RX-bytes: 0

 Stats reg 11 RX-packets: 0          RX-errors: 0          RX-bytes: 0

 Stats reg 12 RX-packets: 0          RX-errors: 0          RX-bytes: 0

 Stats reg 13 RX-packets: 0          RX-errors: 0          RX-bytes: 0

 Stats reg 14 RX-packets: 0          RX-errors: 0          RX-bytes: 0

 Stats reg 15 RX-packets: 0          RX-errors: 0          RX-bytes: 0

 Stats reg 0 TX-packets: 11506      TX-bytes: 3136755

 Stats reg 1 TX-packets: 1967       TX-bytes: 1092416

 Stats reg 2 TX-packets: 3203       TX-bytes: 1055440

 Stats reg 3 TX-packets: 3623       TX-bytes: 1553650

 Stats reg 4 TX-packets: 8502       TX-bytes: 1754202

 Stats reg 5 TX-packets: 2495       TX-bytes: 935701

 Stats reg 6 TX-packets: 3594       TX-bytes: 1334217

 Stats reg 7 TX-packets: 2160       TX-bytes: 728911

 Stats reg 8 TX-packets: 3293       TX-bytes: 1143037

 Stats reg 9 TX-packets: 3445       TX-bytes: 1282575

 Stats reg 10 TX-packets: 3733       TX-bytes: 1419152

 Stats reg 11 TX-packets: 7225       TX-bytes: 1497078

 Stats reg 12 TX-packets: 6806       TX-bytes: 1570908

 Stats reg 13 TX-packets: 3034       TX-bytes: 982978

 Stats reg 14 TX-packets: 5574       TX-bytes: 1535978

 Stats reg 15 TX-packets: 3807       TX-bytes: 1351594

 ############################################################################

 

Lab-Firewall#show nic stats port 0" target node0.fpc0

Aug 15 18:32:36

================ tnp_0x10001081 ================

SENT: Ukern command: show nic stats port 0

 

 ######################## NIC statistics for port 0 ########################

 RX-packets: 488685     RX-errors: 0          RX-bytes: 313134279

 RX-nombuf: 0

  TX-packets: 73967      TX-errors: 0          TX-bytes: 22554478                         <<<<<

 Stats reg 0 RX-packets: 61719      RX-errors: 0          RX-bytes: 30223453

 Stats reg 1 RX-packets: 45396      RX-errors: 0          RX-bytes: 19900315

 Stats reg 2 RX-packets: 37026      RX-errors: 0          RX-bytes: 24100888

 Stats reg 3 RX-packets: 36384      RX-errors: 0          RX-bytes: 13191281

 Stats reg 4 RX-packets: 38980      RX-errors: 0          RX-bytes: 15080392

 Stats reg 5 RX-packets: 44501      RX-errors: 0          RX-bytes: 17099781

 Stats reg 6 RX-packets: 46914      RX-errors: 0          RX-bytes: 30682620

 Stats reg 7 RX-packets: 133412     RX-errors: 0          RX-bytes: 143082867

 Stats reg 8 RX-packets: 43447      RX-errors: 0          RX-bytes: 19697040

 Stats reg 9 RX-packets: 906        RX-errors: 0          RX-bytes: 75642

 Stats reg 10 RX-packets: 0          RX-errors: 0          RX-bytes: 0

 Stats reg 11 RX-packets: 0          RX-errors: 0          RX-bytes: 0

 Stats reg 12 RX-packets: 0          RX-errors: 0          RX-bytes: 0

 Stats reg 13 RX-packets: 0          RX-errors: 0          RX-bytes: 0

 Stats reg 14 RX-packets: 0          RX-errors: 0          RX-bytes: 0

 Stats reg 15 RX-packets: 0          RX-errors: 0          RX-bytes: 0

 Stats reg 0 TX-packets: 11506      TX-bytes: 3136755

 Stats reg 1 TX-packets: 1967       TX-bytes: 1092416

 Stats reg 2 TX-packets: 3203       TX-bytes: 1055440

 Stats reg 3 TX-packets: 3623       TX-bytes: 1553650

 Stats reg 4 TX-packets: 8502       TX-bytes: 1754202

 Stats reg 5 TX-packets: 2495       TX-bytes: 935701

 Stats reg 6 TX-packets: 3594       TX-bytes: 1334217

 Stats reg 7 TX-packets: 2161       TX-bytes: 728957

 Stats reg 8 TX-packets: 3293       TX-bytes: 1143037

 Stats reg 9 TX-packets: 3445       TX-bytes: 1282575

 Stats reg 10 TX-packets: 3733       TX-bytes: 1419152

 Stats reg 11 TX-packets: 7226       TX-bytes: 1497124

 Stats reg 12 TX-packets: 6806       TX-bytes: 1570908

 Stats reg 13 TX-packets: 3034       TX-bytes: 982978

 Stats reg 14 TX-packets: 5574       TX-bytes: 1535978

 Stats reg 15 TX-packets: 3807       TX-bytes: 1351594

 ############################################################################

 

Lab-Firewall#request pfe execute command "show nic stats port 0" target node0.fpc0

Aug 15 18:32:46

================ tnp_0x10001081 ================

SENT: Ukern command: show nic stats port 0

 

 ######################## NIC statistics for port 0 ########################

 RX-packets: 488686     RX-errors: 0          RX-bytes: 313134653

 RX-nombuf: 0

 TX-packets: 73967     TX-errors: 0          TX-bytes: 22554900                           <<<<<

 Stats reg 0 RX-packets: 61720      RX-errors: 0          RX-bytes: 30223827

 Stats reg 1 RX-packets: 45396      RX-errors: 0          RX-bytes: 19900315

 Stats reg 2 RX-packets: 37026      RX-errors: 0          RX-bytes: 24100888

 Stats reg 3 RX-packets: 36384      RX-errors: 0          RX-bytes: 13191281

 Stats reg 4 RX-packets: 38980      RX-errors: 0          RX-bytes: 15080392

 Stats reg 5 RX-packets: 44501      RX-errors: 0          RX-bytes: 17099781

 Stats reg 6 RX-packets: 46914      RX-errors: 0          RX-bytes: 30682620

 Stats reg 7 RX-packets: 133412     RX-errors: 0          RX-bytes: 143082867

 Stats reg 8 RX-packets: 43447      RX-errors: 0          RX-bytes: 19697040

 Stats reg 9 RX-packets: 906        RX-errors: 0          RX-bytes: 75642

 Stats reg 10 RX-packets: 0          RX-errors: 0          RX-bytes: 0

 Stats reg 11 RX-packets: 0          RX-errors: 0          RX-bytes: 0

 Stats reg 12 RX-packets: 0          RX-errors: 0          RX-bytes: 0

 Stats reg 13 RX-packets: 0          RX-errors: 0          RX-bytes: 0

 Stats reg 14 RX-packets: 0          RX-errors: 0          RX-bytes: 0

 Stats reg 15 RX-packets: 0          RX-errors: 0          RX-bytes: 0

 Stats reg 0 TX-packets: 11506      TX-bytes: 3136755

 Stats reg 1 TX-packets: 1967       TX-bytes: 1092416

 Stats reg 2 TX-packets: 3203       TX-bytes: 1055440

 Stats reg 3 TX-packets: 3623       TX-bytes: 1553650

 Stats reg 4 TX-packets: 8502       TX-bytes: 1754202

 Stats reg 5 TX-packets: 2495       TX-bytes: 935701

 Stats reg 6 TX-packets: 3594       TX-bytes: 1334217

 Stats reg 7 TX-packets: 2162       TX-bytes: 729319

 Stats reg 8 TX-packets: 3293       TX-bytes: 1143037

 Stats reg 9 TX-packets: 3445       TX-bytes: 1282575

 Stats reg 10 TX-packets: 3733       TX-bytes: 1419152

 Stats reg 11 TX-packets: 7227       TX-bytes: 1497170

 Stats reg 12 TX-packets: 6806       TX-bytes: 1570908

 Stats reg 13 TX-packets: 3034       TX-bytes: 982978

 Stats reg 14 TX-packets: 5574       TX-bytes: 1535978

 Stats reg 15 TX-packets: 3807       TX-bytes: 1351594

 ############################################################################

In the above example, notice TX-packets are not incrementing. That means DPDK port is in stuck state.

Solution

1, Below is the matching PR for this issue:

https://prsearch.juniper.net/problemreport/PR1493940

Issue is resolved in below Junos versions:

Junos
15.1X49-D240
18.4R3-S5
19.2R3-S1
19.3R3
19.4R3
20.1R2
20.2R2
20.3R1
20.4R1


2, Another PR introduces a way to reset the DPDP once the port is stuck. 
https://prsearch.juniper.net/problemreport/PR1626562

On the SRX4100 and SRX4200 platforms, it can detect the DPDK (data plane development kit) Tx stuck issue and trigger a major chassis alarm goes which might trigger RG1 failover to the healthy node. A DPDK reset will be triggered only to the stuck port and if the reset resolves the tx stuck issue, the major chassis alarm will go off.

Change introduced in

Junos
19.4R3-S8
19.4R3-S9
20.2R3-S7
20.4R3-S4
21.2R3-S1
21.2R3-S3
21.4R3-S3
22.2R1
22.2R2
22.3R1
22.4R1

Modification History

2023-10-13