Input errors were increasing on 100G ports of PTX10001-36MR running EVO release 21.2R3-S1.6-EVO.
1.) Device is PTX10001-36MR running EVO release 21.2R3-S1.6-EVO.
2.) Input errors were increasing on 100G ports et-0/0/1 & et-0/0/2.
Only the Total errors counter was incrementing. There were no other counters reporting increments.
user@device> show interfaces et-0/0/1 extensive |match "error|drop"
<>
Errors: 134, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0
Total errors 134 0
f69466@NTT8F-BR-P10K1-002-RE0> show interfaces et-0/0/2 extensive |match "error|drop"
Errors: 178, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0
Total errors 178 0
Initial Action Plan:
1.) Customer to help us with the details of the peer side connections (device, optics, SMF/MMF, etc) for ports et-0/0/1 & et-0/0/2.
2.) Customer to capture the below outputs from the devices.
i. Clear counters on the device for baselines
> set cli timestamp
> clear interfaces statistics et-0/0/1
> clear interfaces statistics et-0/0/2
> clear pfe statistics traffic
> start shell pfe network fpc0
pfe:> clear cda trapstats
ii. Capture the counter values after the clear, for baselines:
> show interfaces et-0/0/1 extensive |match "error|drop"
> show interfaces et-0/0/2 extensive |match "error|drop"
> show pfe statistics traffic
pfe:> show cda trapstats
pfe:> show pfe statistics traffic
iii. Wait for 10-12 hours and capture another set of values.
Observations:
As per the analysis, the total error count across the two interfaces is found to be 312 (134+178).
The pkt_eope exception on the router has a matching count of 312.
f69466@NTT8F-BR-P10K1-002-RE0:pfe> show cda trapstats
0 100 slu.trapcode.pkt_eope 312 0
Capturing the exception packets:
1.) Obtain the TRAPCODE number for pkt_eope. It should be 100 as per shared output.
Reference Output:
user@device> start shell pfe network fpc0
user@device:pfe> show cda trapstats | grep pkt_eope
DEV TRAPCODE NAME COUNT RATE
user@device:pfe> exit
2.) Enter the RE shell as root user.
user@device> start shell user root
[vrf:none] user@device-RE0:~#
3.) Run the below command from RE shell as root. This built-in packet collector will run in the background and collect 10 packets from pfe 0 matching trapcode 100. The packets will be written in /var/tmp with filename pkt.eope.txt.
[vrf:none] user@device-RE0:~# parcel_dump.py -c 10 -p 0 -t 100 > /var/tmp/pkt-eope.txt 2>&1 &
4.) Validate that the parcel_dump script is running in the background.
[vrf:none] user@device:~# ps aux | grep parcel
root 5761 2.0 0.0 22012 19668 pts/0 S 12:39 0:00 python3 /usr/sbin/parcel_dump.py -c 10 -p 0 -t 100
root 6199 0.0 0.0 4668 2804 pts/0 S+ 12:39 0:00 grep parcel
5.) Leave the script running for a few hours. The script will automatically terminate once 10 exception packets are captured.
root 8905 0.0 0.0 4668 2780 pts/0 S+ 12:40 0:00 grep parcel
6.) Upload the pkt-eope.txt file from /var/tmp/.
The issue was isolated to a mismatch in the mtu configuration on both sides.
The mtu was set to 1592 on the IFD and 1500 on the IFL's.
The customer tuned the mtu values on the peer side to match the local values.
This resolved the issue and the error counters stopped incrementing.