On PTX platforms, a port may not come up (physical link is down) after an interface flap, reboot of the line card, or transport gear restart on the remote end.
Topology:
PTX5000-A : et-1/0/8 < Transport device > PTX500o-B: et-8/1/0 is down.
Description
Model: ptx5000 Junos: 17.4X6.15 FPC1: FPC-P2 xcvr: QSFP-100G-CWDM4
Physical interface: et-1/0/8, Enabled, Physical link is Down Interface index: 282, SNMP ifIndex: 586, Generation: 285 Link-level type: Ethernet, MTU: 1514, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled Device flags : Present Running Down Interface flags: Hardware-Down SNMP-Traps Internal: 0x4000 Link flags : None CoS queues : 8 supported, 8 maximum usable queues Hold-times : Up 0 ms, Down 0 ms Damping : half-life: 0 sec, max-suppress: 0 sec, reuse: 0, suppress: 0, state: unsuppressed Current address: 38:4f:49:7c:d9:59, Hardware address: 38:4f:49:7c:d9:59 Last flapped : 2020-11-30 20:36:59 UTC (14:50:16 ago) Statistics last cleared: 2020-11-30 22:00:42 UTC (13:26:33 ago) Traffic statistics: Input bytes : 11121699880758 0 bps Output bytes : 11121720347341 0 bps Input packets: 2446480842 0 pps Output packets: 2446539907 0 pps IPv6 transit statistics: Input bytes : 0 Output bytes : 0 Input packets: 0 Output packets: 0 Input errors: Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0 Output errors: Carrier transitions: 0, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0 Egress queues: 8 supported, 5 in use Queue counters: Queued packets Transmitted packets Dropped packets 0 0 0 0 1 0 0 0 2 0 0 0 3 0 0 0 5 0 0 0 Queue number: Mapped forwarding classes 0 FC-BEST-EFFORT, FC-STREAMING 1 FC-LOW-PRIORITY-DATA 2 FC-LOW-LATENCY-DATA, FC-SIGNALING, FC-REAL-TIME 3 FC-NETWORK-CONTROL 5 FC-TELEPHONY Active alarms : LINK Active defects : LINK, LOCAL-FAULT PCS statistics Seconds Bit errors 0 Errored blocks 0 Ethernet FEC Mode : FEC91 Ethernet FEC statistics Errors FEC Corrected Errors 0 FEC Uncorrected Errors 0 FEC Corrected Errors Rate 0 FEC Uncorrected Errors Rate 0 MAC statistics: Receive Transmit Total octets 0 0 Total packets 0 0 Unicast packets 0 0 Broadcast packets 0 0 Multicast packets 0 0 CRC/Align errors 0 0 FIFO errors 0 0 MAC control frames 0 0 MAC pause frames 0 0 Oversized frames 0 Jabber frames 0 Fragment frames 0 VLAN tagged frames 0 Code violations 0 Filter statistics: Input packet count 0 Input packet rejects 0 Input DA rejects 0 Input SA rejects 0 Output packet count 0 Output packet pad count 0 Output packet error count 0 CAM destination filters: 0, CAM source filters: 0 Packet Forwarding Engine configuration: Destination slot: 0 (0x00) CoS information: Direction : Output CoS transmit queue Bandwidth Buffer Priority Limit % bps % usec 0 FC-BEST-EFFORT, FC-STREAMING 95 95000000000 95 0 low none 3 FC-NETWORK-CONTROL 5 5000000000 5 0 low none Link Degrade : Link Monitoring : Disable Interface transmit statistics: Disabled
Log messages
Dec 1 05:11:38 PTX5000-A : %PFE-3: fpc1 t6e_dfe_tuning_state:et-1/0/8 - Failed to dfe tuning count 10 Dec 1 05:30:29 PTX5000-A : %PFE-3: fpc1 t6e_dfe_tuning_state:et-1/0/8 - Failed to dfe tuning count 10 Dec 1 05:49:21 PTX5000-A : %PFE-3: fpc1 t6e_dfe_tuning_state:et-1/0/8 - Failed to dfe tuning count 10 Dec 1 06:08:13 PTX5000-A : %PFE-3: fpc1 t6e_dfe_tuning_state:et-1/0/8 - Failed to dfe tuning count 10 Dec 1 06:27:07 PTX5000-A : %PFE-3: fpc1 t6e_dfe_tuning_state:et-1/0/8 - Failed to dfe tuning count 10 Dec 1 06:46:01 PTX5000-A : %PFE-3: fpc1 t6e_dfe_tuning_state:et-1/0/8 - Failed to dfe tuning count 10 Dec 1 07:04:52 PTX5000-A : %PFE-3: fpc1 t6e_dfe_tuning_state:et-1/0/8 - Failed to dfe tuning count 10 Dec 1 07:23:46 PTX5000-A : %PFE-3: fpc1 t6e_dfe_tuning_state:et-1/0/8 - Failed to dfe tuning count 10 Dec 1 07:42:38 PTX5000-A : %PFE-3: fpc1 t6e_dfe_tuning_state:et-1/0/8 - Failed to dfe tuning count 10 Dec 1 08:01:31 PTX5000-A : %PFE-3: fpc1 t6e_dfe_tuning_state:et-1/0/8 - Failed to dfe tuning count 10
On the PTX5000/PTX3000 platforms with t6e-pic installed (e.g. P3-10-U-QSFP28, P3-15-U-QSFP28, P3-24-U-QSFP28), the interface may fail to perform DFE tuning after link flaps on those PICs. Because of this, the interface may be stuck in down status. The issue is that under some instances, t6e tuning code can bypass retimer false pmd-lock check. This could happen due to one of the following:
Repetitive requests to start tuning after Avago serdes failed to respond to previous stop tuning request.
By enabling and adding debug logs, it revealed the existing serdes recovery mechanism (serdes shutdown + serdes re-init) in the PIC driver was not getting executed due to not meeting the required conditions even after a long wait time (approx. 1-2 hours of wait).
The issue is fixed in Junos 17.4X6.20 and later releases.
The following workaround can be used if you are running a release earlier than 17.4X6.20 and upgrade is not possible. Please contact your JTAC Representative to assist with the restoration steps.
Restoration steps:
Keep the hard loop for 15 minutes check if link comes UP.
Collect the script output, session logs, /var/log/messages. Check if any serdes lane is busy.
If busy, issue the following commands:
vty fpc <fpc_num> set parser security 10 test t6e-pic select <t6e_pic_number> test t6e-pic serdes-repair <port_num> <chan_num> 2
Wait for at least 15 minutes to allow DFE tuning to complete and see if the link comes UP. If not, try step 2 again.
If recovery does not help, contact your JTAC Representative to assist with manual recovery.