Description

Customer tried to physically cutover 4x10G Internet links from Cisco N6K to new Juniper MX304, but the the links remained in "down" state after.

Symptoms

The 10G link on a new MX304 failed be up. Customer tried with local loop, but the physical link was still down with "Local-Fault".

set interfaces xe-0/0/4:0 gigether-options loopback

commit

 

RE0> show interfaces xe-0/0/4:0 

Physical interface: xe-0/0/4:0, Enabled, Physical link is Down

 Interface index: 180, SNMP ifIndex: 537

 Description: via tap

 Link-level type: Ethernet, MTU: 1514, MRU: 1522, LAN-PHY mode, Speed: 10Gbps,

 BPDU Error: None, Loop Detect PDU Error: None, MAC-REWRITE Error: None,

 Loopback: Local, Source filtering: Disabled, Flow control: Disabled,

 Speed Configuration: Auto

 Device flags  : Present Running Down Loop-Detected

 Interface flags: Hardware-Down SNMP-Traps Internal: 0x4000

 Link flags   : None

 CoS queues   : 8 supported, 8 maximum usable queues

 Schedulers   : 0

 Current address: 98:49:25:46:14:16, Hardware address: 98:49:25:46:14:16

 Last flapped  : 2024-03-27 14:30:33 AEDT (6d 05:39 ago)

 Input rate   : 0 bps (0 pps)

 Output rate  : 0 bps (0 pps)

 Active alarms : LINK

 Active defects : LINK, LOCAL-FAULT

 PCS statistics           Seconds

  Bit errors               7

  Errored blocks             7

Solution

Checking in the pfe shell, found the port was stuck in "Init_Done" state

 

fpc0:pfe> show picd channel summary 

 

channel_table :

                            pfe   port  xcvr  rx   pfe

name        speed  chan_state  serdes_state  portE  bindE  rx   soak  inst  stream

----------------  -----  -----------  ------------  -----  -----  ----  ----  ----  ------

channel-0/0/0:0  100G  Chan_Online  Tune_Ok    up   up   up   up    1    0

channel-0/0/1:0  100G  Chan_Online  Tune_Ok    up   up   up   up    1    4

channel-0/0/2:0  100G  Chan_Online  Tune_Ok    up   up   up   up    1    8

channel-0/0/3:0  100G  Chan_Online  Tune_Ok    up   up   up   up    1    12

channel-0/0/4:0  10G   Chan_Online  Init_Done   up   down  down  down   1    8

channel-0/0/5:0  10G   Chan_Online  Init_Done   up   down  up   up    1    12

channel-0/0/6:0  100G  Chan_Online  Tune_Ok    up   up   up   up    1    0

channel-0/0/7:0  100G  Chan_Online  Tune_Ok    up   up   up   up    1    4

channel-0/0/8:0  100G  Chan_Online  Tune_Ok    up   up   up   up    0    12

channel-0/0/9:0  100G  Chan_Online  Tune_Ok    up   up   up   up    0    8

channel-0/0/10:0  100G  Chan_Online  Tune_Ok    up   up   up   up    0    4

channel-0/0/11:0  100G  Chan_Online  Tune_Ok    up   up   up   up    0    0

channel-0/0/12:0  100G  Chan_Online  Tune_Ok    up   up   up   up    0    4

channel-0/0/13:0  100G  Chan_Online  Tune_Ok    up   up   up   up    0    0

channel-0/0/14:0  10G   Chan_Online  Tune_Ok    up   down  down  down   0    14

channel-0/0/15:0  10G   Chan_Online  Init_Done   up   down  down  down   0    10

 

Customer mentioned the "problem" ports were tested with 100G optics in the past.

 

There is a bug(PR1781563) in JUNOS recently that the above reported issue may happen when user re-configure a port to a different speed mode that involves a replacement of XCVR, following a non-preferable sequence that:
A. Prior to changing the port speed configuration, replace the XCVR with a new type that does not support the old port speed. This will potentially make the XCVR driver SW put the new XCVR into a Fault state due to DP init failure.
B. After inserting a new XCVR, user configure the port speed mode that is supported by the new XCVR. However, XCVR SW no longer accept the new configuration since the XCVR object has been in a Fault state.

The workaround is to reboot the FPC.
After that, this customer's problem was cleared.

To avoid such bug, it would be better to change the port speed configure prior to replacing the XCVR.

Modification History

2024-04-04 : Article Created