Hardware input drops were seen on MX240
Router> show pfe statistics trafficPacket Forwarding Engine local traffic statistics: Local packets input : 1481420345 Local packets output : 6256572012 Software input control plane drops : 0 Software input high drops : 0 Software input medium drops : 0 Software input low drops : 0 Software output drops : 0 Hardware input drops : 84732956 -------------->>> Drops
1. Initally collect below RE and FPC shell command output from problematic router. This will help you to know if there is any drop due to ddos, or between FPC's. ========= Initial Troubelshooting ========== ******* RE CLI Commands ******** show ddos-protection protocols statistics | no-more show class-of-service fabric statistics | no-more show pfe statistics traffic | no-more ----------------->> This will show output of all the FPC. show pfe statistics traffic fpc 0 | no-more ---------->> Please collect FPC specific output for all the FPC's to know which all FPC's are contributing to the drops. Below is the couple of example for different FPC's show pfe statistics traffic fpc 1 | no-more show pfe statistics traffic fpc 2 | no-more ******* FPC Shell Commands ******** request pfe execute target fpc0 command "show xmchip 0 q-node all-stats 0" | no-more request pfe execute target fpc0 command "show xmchip 0 q-node all-stats 1" | no-more request pfe execute target fpc0 command "show xmchip 0 phy-stream list 0" | no-more request pfe execute target fpc0 command "show xmchip 0 q-node list 0" | no-more request pfe execute target fpc0 command "show jnh 0 exceptions" | no-more | refresh 20 2. We saw below output under "show pfe statistics traffic" and "show pfe statistics traffic fpc 0". This shows issue is with FPC 0 Router> show pfe statistics traffic Packet Forwarding Engine local traffic statistics: Local packets input : 1600207116 Local packets output : 6810504960 Software input control plane drops : 0 Software input high drops : 0 Software input medium drops : 0 Software input low drops : 0 Software output drops : 0 Hardware input drops : 116118979 Router> show pfe statistics traffic fpc 0 | no-more Packet Forwarding Engine local traffic statistics: Local packets input : 1244699296 Local packets output : 5845045494 Software input control plane drops : 0 Software input high drops : 0 Software input medium drops : 0 Software input low drops : 0 Software output drops : 0 Hardware input drops : 116118979 Router> show pfe statistics traffic fpc 1 | no-more Packet Forwarding Engine local traffic statistics: Local packets input : 242233458 Local packets output : 502783356 Software input control plane drops : 0 Software input high drops : 0 Software input medium drops : 0 Software input low drops : 0 Software output drops : 0 Hardware input drops : 03. "show xmchip 0 q-node all-stats 0" command will list all the queues starting from "Queue statistics (Queue 0000) to (Queue 1015)". You should see drops in this. Below was the output from the problematic router. Queue statistics (Queue 1014) ----------------------------- -------------------------------------------------------------------------------------------- Color Outcome Counter Counter Total Rate Index Name -------------------------------------------------------------------------------------------- All Forwarded (No rule) 456 Packets 0 0 pps All Forwarded (No rule) 456 Bytes 0 0 bps All Forwarded (Rule) 457 Packets 378715213 35 pps All Forwarded (Rule) 457 Bytes 93425388946 27528 bps All Force drops 460 Packets 0 0 pps All Force drops 460 Bytes 0 0 bps All Error drops 461 Packets 0 0 pps All Error drops 461 Bytes 0 0 bps 0 WRED drops 458 Packets 67524402 0 pps 0 WRED drops 458 Bytes 52317876079 0 bps 0 TAIL drops 459 Packets 48594577 0 pps 0 TAIL drops 459 Bytes 35349441477 0 bps 1 WRED drops 462 Packets 0 0 pps 1 WRED drops 462 Bytes 0 0 bps 1 TAIL drops 463 Packets 0 0 pps 1 TAIL drops 463 Bytes 0 0 bps 2 WRED drops 464 Packets 0 0 pps 2 WRED drops 464 Bytes 0 0 bps 2 TAIL drops 465 Packets 0 0 pps 2 TAIL drops 465 Bytes 0 0 bps 3 WRED drops 466 Packets 0 0 pps 3 WRED drops 466 Bytes 0 0 bps 3 TAIL drops 467 Packets 0 0 pps 3 TAIL drops 467 Bytes 0 0 bps --------------------------------------------------------------------------------------------4. Problem is seen in Queue 1014 and with the help of below command you can find a physical stream for this queue. Each physical stream has 8 queues.In below output physical stream: 1151 there are 8 queues (0 to 7). Where each queue represent a queue node like queue 0 repesents queue node 1008 and so on.Hence queue 6 repesents queue node 1014. request pfe execute target fpc0 command "show xmchip 0 q-node list 0" | no-more Queue scheduler node list ------------------------- ---------------------------------------------- Queue node L2 node L1 node PHY stream ---------------------------------------------- 1008 252 126 1151 1009 252 126 1151 1010 252 126 1151 1011 252 126 1151 1012 252 126 1151 1013 252 126 1151 1014 252 126 1151 1015 252 126 1151 5. To confirm above is the only problem area, we requested to clear the PFE statistics and shared below script to customer. They kept it running on the router for couple of days. You can modify the script and add more commands or change FPC number depending on your case. clear pfe statistics traffic start shell user root cat > Sript_Hardware_input_drops.sh while [ : ] do date echo "=====show pfe statistics traffic=====" cli -c "show pfe statistics traffic | no-more" echo "=====show pfe statistics traffic fpc 0=====" cli -c "show pfe statistics traffic fpc 0 | no-more" echo "=====show pfe statistics traffic fpc 1=====" cli -c "show pfe statistics traffic fpc 1 | no-more" echo "=====show ddos-protection protocols statistics=====" cli -c "show ddos-protection protocols statistics | no-more" echo "=====cprod -A fpc0 -c "show pfe statistics traffic"=====" cprod -A fpc0 -c "show pfe statistics traffic" echo "=====cprod -A fpc0 -c "show xmchip 0 phy-stream q-node-stats 1151 6"=====" cprod -A fpc0 -c "show xmchip 0 phy-stream q-node-stats 1151 6" echo "=====cprod -A fpc0 -c "show ddos policer stats resolve"=====" cprod -A fpc0 -c "show ddos policer stats resolve" echo "=====cprod -A fpc0 -c "show ddos policer stats all"=====" cprod -A fpc0 -c "show ddos policer stats all" sleep 300 done Use below command to execute the script and save the data under /var/tmp. Script with keep on running in background and you can ask customer to periodically check RE cli command output of "show pfe statistics traffic" to see if you see hardware input drops incrementing. sh Sript_Hardware_input_drops.sh > /var/tmp/Data_Hardware_input_output.txt &6. Once you saw the drops counter incrementing under "show pfe statistics traffic" then ask customer to share the data file "Data_Hardware_input_output.txt" from /var/tmp.7. Below the output which we got from customer. +++++++++++ 1st set of data after clearing the pfe statistic traffic +++++++++++++ Wed Feb 28 00:19:45 PST 2024 =====show pfe statistics traffic===== Packet Forwarding Engine local traffic statistics: Local packets input : 972423 Local packets output : 3775749 Software input control plane drops : 0 Software input high drops : 0 Software input medium drops : 0 Software input low drops : 0 Software output drops : 0 Hardware input drops : 0 --------->> Counter reset to zero =====show pfe statistics traffic fpc 0===== Packet Forwarding Engine local traffic statistics: Local packets input : 616956 Local packets output : 2838500 Software input control plane drops : 0 Software input high drops : 0 Software input medium drops : 0 Software input low drops : 0 Software output drops : 0 Hardware input drops : 0 --------->> Counter reset to zero =====cprod -A fpc0 -c show xmchip 0 phy-stream q-node-stats 1151 6===== Queue statistics (Queue 1014) ----------------------------- -------------------------------------------------------------------------------------------- Color Outcome Counter Counter Total Rate Index Name -------------------------------------------------------------------------------------------- All Forwarded (No rule) 456 Packets 0 0 pps All Forwarded (No rule) 456 Bytes 0 0 bps All Forwarded (Rule) 457 Packets 392915373 19 pps All Forwarded (Rule) 457 Bytes 97617646267 16384 bps All Force drops 460 Packets 0 0 pps All Force drops 460 Bytes 0 0 bps All Error drops 461 Packets 0 0 pps All Error drops 461 Bytes 0 0 bps 0 WRED drops 458 Packets 71857709 0 pps --------->> Final counter after clearing 0 WRED drops 458 Bytes 55674770653 0 bps 0 TAIL drops 459 Packets 51453775 0 pps --------->> Final counter after clearing 0 TAIL drops 459 Bytes 37434688731 0 bps 1 WRED drops 462 Packets 0 0 pps 1 WRED drops 462 Bytes 0 0 bps 1 TAIL drops 463 Packets 0 0 pps 1 TAIL drops 463 Bytes 0 0 bps 2 WRED drops 464 Packets 0 0 pps 2 WRED drops 464 Bytes 0 0 bps 2 TAIL drops 465 Packets 0 0 pps 2 TAIL drops 465 Bytes 0 0 bps 3 WRED drops 466 Packets 0 0 pps 3 WRED drops 466 Bytes 0 0 bps 3 TAIL drops 467 Packets 0 0 pps 3 TAIL drops 467 Bytes 0 0 bps -------------------------------------------------------------------------------------------- +++++++++++ Last set of data after running the script +++++++++++++ Fri Mar 1 02:01:17 PST 2024 =====show pfe statistics traffic===== Packet Forwarding Engine local traffic statistics: Local packets input : 14262172 Local packets output : 61520431 Software input control plane drops : 0 Software input high drops : 0 Software input medium drops : 0 Software input low drops : 0 Software output drops : 0 Hardware input drops : 4179836 --------->> Counter incremented =====show pfe statistics traffic fpc 0===== Packet Forwarding Engine local traffic statistics: Local packets input : 10082618 Local packets output : 51002178 Software input control plane drops : 0 Software input high drops : 0 Software input medium drops : 0 Software input low drops : 0 Software output drops : 0 Hardware input drops : 4179836 --------->> Counter incremented on FPC 0 =====cprod -A fpc0 -c show xmchip 0 phy-stream q-node-stats 1151 6===== Queue statistics (Queue 1014) ----------------------------- -------------------------------------------------------------------------------------------- Color Outcome Counter Counter Total Rate Index Name -------------------------------------------------------------------------------------------- All Forwarded (No rule) 456 Packets 0 0 pps All Forwarded (No rule) 456 Bytes 0 0 bps All Forwarded (Rule) 457 Packets 399040549 1 pps All Forwarded (Rule) 457 Bytes 99940235082 1392 bps All Force drops 460 Packets 0 0 pps All Force drops 460 Bytes 0 0 bps All Error drops 461 Packets 0 0 pps All Error drops 461 Bytes 0 0 bps 0 WRED drops 458 Packets 74197202 0 pps --------> Counter incremented <<< 74197202 – 71857709 = 2339493 0 WRED drops 458 Bytes 57486259771 0 bps 0 TAIL drops 459 Packets 53294118 0 pps --------> Counter incremented <<< 53294118 – 51453775 = 1840343 0 TAIL drops 459 Bytes 38780006394 0 bps 1 WRED drops 462 Packets 0 0 pps 1 WRED drops 462 Bytes 0 0 bps 1 TAIL drops 463 Packets 0 0 pps 1 TAIL drops 463 Bytes 0 0 bps 2 WRED drops 464 Packets 0 0 pps 2 WRED drops 464 Bytes 0 0 bps 2 TAIL drops 465 Packets 0 0 pps 2 TAIL drops 465 Bytes 0 0 bps 3 WRED drops 466 Packets 0 0 pps 3 WRED drops 466 Bytes 0 0 bps 3 TAIL drops 467 Packets 0 0 pps 3 TAIL drops 467 Bytes 0 0 bps -------------------------------------------------------------------------------------------- Total WRED and TAIL Drops = 2,339,493 + 1,840,343 = 4,179,836 Hardware input drops in PFE stats = 4,179,8368. The Queue 6 belongs to the following protocols. The good part is that there is no control plane traffic and it is only ICMP error handling traffic which should be MTU exceeded, Resolve NH or Reject. There should not be any traffic impact due to it. =====show ddos asic punt-proto-maps===== PUNT exceptions directly mapped to DDOS proto: code PUNT name group proto pid q# bwidth burst ---- -------------------- --------- ------ ---- -- ------ ------ 40 PUNT_REJECT reject aggregate 4600 6 2000 10000 68 PUNT_REJECT_V6 rejectv6 aggregate 5900 6 2000 10000 DISCARD exceptions mapped to DDOS proto: code DISC name group proto pid q# bwidth burst ---- -------------------- --------- ------ ---- -- ------ ------ 0 All Other Discards exception unclass.. 4001 6 250 250 21 MTU_EXCEEDED exception mtu-exceed 4002 6 250 250 8 ENUM_CHK_MISMATCH exception mcast-rpf 4003 6 250 250 PUNT exceptions parsed by their own parsers ------------------------------------------------------------------ resolve other 101 6 2000 2000 resolve ucast-v4 102 6 3000 5000 resolve mcast-v4 103 6 3000 5000 resolve ucast-v6 104 6 3000 5000 resolve mcast-v6 105 6 3000 5000 REJECT_FW exception mapped to DHCPv4/6 and filter-act. Only filter-act shown ------------------------------------------------------------------ filter-act other 201 6 2000 5000 filter-act filter-v4 202 6 2000 5000 filter-act filter-v6 203 6 2000 50009. Share the analysis with Customer and you can request for closure.Note: This is initial procedure to start troubelshooting of hardware input drops. You might not see similar symptoms but this can help you to get started.