Description

QSFP of type QSFP-100GBASE-LR4 on a MPC11 (MPC11E 3D MRATE-40xQSFPP) card failed after some "module high temperature detected" and "i2c transaction error" log messages appeared.

Symptoms

Sep 1 06:45:03.729 MX2020_RE0-fpc13 picd: PICD_MODULE_ALARM: xcvr-13/0/2 Module high temperature detected 

Sep 1 06:45:03.528 MX2020_RE0 kernel: i2c i2c-28: (3/0:0x50) i2c transaction error (0x00000006)

Sep 1 06:45:04.016 MX2020_RE0 kernel: i2c i2c-28: (3/0:0x50) i2c transaction error (0x00000006)

 

QSFP not getting detected:

 

FPC 13      REV 50  750-086583  EBAE3668     MPC11E 3D MRATE-40xQSFPP

 CPU      REV 21  750-072571  EBAD7041     FMPC PMB

 PIC 0          BUILTIN   BUILTIN      MRATE-5xQSFPP

  Xcvr 0    REV 01  740-058732  1W1CQAA5422T6   QSFP-100GBASE-LR4

  Xcvr 1    REV 01  740-058732  1W1CQAA5422NR   QSFP-100GBASE-LR4

  Xcvr 2    0    NON-JNPR            UNKNOWN

  Xcvr 3    REV 01  740-058732  1W1CQAA5422NM   QSFP-100GBASE-LR4

 

root@MX2020_RE0-fpc13:pfe> show picd optics pic_slot 0 port 2 cmd identifier

 

 PICD optics identifier

 Xcvr Name    :xcvr-13/0/2

 Xcvr Num     :2

 

 *** Identifier information ***

 Identifier      :Unknown

 Connector type    :Unknown

 Encoding       :Unspecified

 Vendor name     :

 

Increment read/write failures and errors:

PICD optics error
  xcvr-13/0/2
  Read Success counters   :35
  Write Success counters  :12
  Read access failures    :583
  Write access failures   :66
  Read recovered errors   :0
  Write recovered errors  :0
  Read err alarm counter  :187
  Write err alarm counter  :22

 PICD optics error

 xcvr-13/0/2

 Read Success counters  :43

 Write Success counters :13

 Read access failures  :737

 Write access failures  :77

 Read recovered errors  :0

 Write recovered errors :0

 Read err alarm counter :341

 Write err alarm counter :33

 

 

Solution

At normal operation time, the software actively monitors the temperature of the XCVR devices. In case a high temperature is detected cross its default threshold, ALARM messages will be logged. In case the temperature crosses its fire-threshold, the concerned XCVR device will be powered off.

 

Information to be collected:

 

start shell pfe network fpc13

show picd optics pic_slot 0 port 2 cmd info

show picd optics pic_slot 0 port 2 cmd diagnostics

show picd optics pic_slot 0 port 2 cmd error

show picd optics pic_slot 0 port 2 cmd alarm

 

Troubleshooting steps:

 

Tried to perform EEPROM rescan:

root@MX2020_RE0-fpc13:pfe> test picd optics cmd eeprom_rescan fpc_slot 13 pic_slot 0 port 2

 PICD optics eeprom_rescan

 xcvr-13/0/2:EEPROM rescan triggered. Wait for 5s

 

Reseat the QSFP and check if it comes up.

If the issue still persists, try to swap the QSFP with a known good QSFP and check if it comes up.

If the new QSFP comes online, proceed with the replacement of the QSFP.

If the new QSFP encounters the same issue, restart the FPC to clear any hung I2C connection issues and replace the FPC if the QSFP fails even after the FPC restart.

 

Modification History

2023-10-20 : Article Created