Description

After commit change or interface bouncing, the port with 400G-ZR-M became down and the typical error log displays with " Module StateMachine Failed " symptom.

Symptoms

The corresponding error logs show up as below and the state on this optic became "Fault" in the PICD outputs.

------------------------

Feb 16 15:01:41.068 2024 noname mib2d[9740]: SNMP_TRAP_LINK_DOWN: ifIndex 515, ifAdminStatus up(1), ifOperStatus down(2), ifName et-0/0/0

Feb 16 15:01:42.048 2024 noname picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/0 chan 0  Rx CDR LOL      alarm set

Feb 16 15:02:32.040 2024 noname picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/0 chan 0  Rx CDR LOL      alarm clear

Feb 16 15:02:40.253 2024 noname mgd[30592]: UI_CMDLINE_READ_LINE: User 'root', command 'set interfaces et-0/0/0 ether-options 802.3ad ae50 '

Feb 16 15:02:46.085 2024 noname picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/0 chan 0  Tx Power Low     alarm set

Feb 16 15:02:46.085 2024 noname picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/0 chan 0  Tx Power Warn Low  alarm set

Feb 16 15:02:50.912 2024 noname picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/0 chan 0  Tx Power Low     alarm clear

Feb 16 15:02:50.912 2024 noname picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/0 chan 0  Tx Power Warn Low  alarm clear

Feb 16 15:02:54.224 2024 noname picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/0 chan 0  Tx Power Low     alarm set

Feb 16 15:02:54.224 2024 noname picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/0 chan 0  Tx Power Warn Low  alarm set

Feb 16 15:02:59.976 2024 noname picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/0 chan 0  Tx Power Warn High  alarm set

Feb 16 15:03:01.023 2024 noname picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/0 chan 0  Tx Power Warn High  alarm clear

Feb 16 15:03:11.022 2024 noname picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/0 chan 0  Tx LASER Fault    alarm set

Feb 16 15:03:11.184 2024 noname mgd[30592]: UI_CMDLINE_READ_LINE: User 'root', command 'show interfaces et-0/0/0 '

Feb 16 15:03:12.021 2024 noname picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/0 chan 0  Tx LASER Fault    alarm clear

Feb 16 15:03:33.017 2024 noname picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/0 chan 0  Rx CDR LOL      alarm set

Feb 16 15:03:37.013 2024 noname picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/0 chan 0  Tx Power Warn High  alarm set

Feb 16 15:03:39.874 2024 noname picd: PICD_MODULE_ALARM: xcvr-0/0/0 Module StateMachine failed

------------------------

root@noname:pfe> show picd xcvr summary 

PICD_OpticsList :

xcvr     pres  cable     Juniper   Rx   Rx   HW init   module

name     ence  type     PN      LOS   LOL   state    state machine

-----------  ----  -----------  ----------  -----  -----  ---------  -------------

xcvr-0/0/0  1   Fault     Fault    Fault  Fault  Fault    Fault

------------------------

show chassis hardware detail no-forwarding

       Hardware inventory:

Item            Version Part number Serial number    Description

Chassis                               XXXXX            JNP10001-36MR [PTX10001-36MR]

<snipped>

FPC 0                    BUILTIN     BUILTIN          FPC-JNP10001-36MR

 PIC 0                  BUILTIN     BUILTIN          8X400GE-MR + 4X100GE-MR

   Xcvr 0      REV 01  740-131169  1T1TZFA725052     QSFP56-DD-400G-ZR-M <<<

-------------------

Solution

To troubleshoot the root cause of this issue:

Please help collect the following when it meets this issue state with JNPR optics.

 

One. CLI outputs.

------------------

request support informaion | save /var/tmp/rsi-noname.log

request system debug-info mode lite

file archive compress source /var/log destination /var/tmp/log

show system commit

show system rollback compare X Y
show chassis pic fpc-slot 0 pic-slot 0

------------------

 

Two. PFE SHELL outputs.

 

------------------

Please replace "x" with the target fpc, pic and port number

start shell pfe network fpc0

show syslog all | no-more

show interfaces et-x/x/x

show chassis errors active 

show mtip-mac summary  

show picd config 

show picd channel summary

show picd channel summary ifd-status

show picd channel by-name chan_name channel-x/x/x:0

show picd serdes fpc x pic x port x chan x 

show picd serdes summary 

show picd link-history fpc x pic x port x chan 0 

show picd port fpc x pic x port x 

show picd link-history fpc x pic x port x chan 0

show picd mac fpc x pic x port x chan 0 

show picd pcs fpc x pic x port x chan 0 

show picd optics fpc_slot x pic_slot x port x cmd alarm

show picd optics fpc_slot x pic_slot x port x cmd diagnostics 

show picd optics fpc_slot x pic_slot x port x cmd dump_devdb_info 

show picd optics fpc_slot x pic_slot x port x cmd dump_xi_info  

show picd optics fpc_slot x pic_slot x port x cmd error  

show picd optics fpc_slot x pic_slot x port x cmd fec_status

show picd optics fpc_slot x pic_slot x port x cmd identifier

show picd optics fpc_slot x pic_slot x port x cmd info 

show picd xcvr fpc x pic x port x cmd info 

show picd xcvr fpc x pic x port x cmd identifier 

show picd xcvr summary  

show picd channel summary ifd-status

test picd optics cmd eeprom_rescan fpc_slot x pic_slot x port x

show picd optics fpc_slot x pic_slot x port x cmd info 

------------------


Three.  please check the details on PICD info as below which should be executed after this command "test picd optics cmd eeprom_rescan fpc_slot 0 pic_slot 0 port 0  " 

-----------------
show picd optics cmd info fpc_slot 0 pic_slot 0 port 0 
 
  PICD optics info
   fpc_num:         0
   pic_num:         0
   port_num:        0
   run_periodic:    true
   periodic_ticks:  98416
   is_diagnostics:  false
   _qsfp_presence:  true
   is_channelized:  false
   config_mismatch: false
   Module firmware revision: 61.22
   Module HW revision: 1.2
   CMIS version:          0x41
   Module State Machine: module_fault
   DP State Machine
     DP 0:    dp_pwrup_wait
   
   EEPROM details for XCVR: xcvr-0/0/0
   QSFP-DD Lower Page:
       0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
  0x00:18 41 04 08 ff 00 00 00 04 00 00 00 00 00 3e 00
  0x10:7d 89 00 00 32 99 00 00 00 00 00 00 00 00 00 00
  0x20:00 00 00 00 03 00 00 3d 16 00 00 00 00 00 00 00
  0x30:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x40:1e f3 f7 f5 00 00 00 00 00 00 00 00 00 00 00 00
  0x50:00 00 00 00 00 02 11 3e 81 01 11 3f 81 01 0d 3e
  0x60:21 55 11 c5 81 01 11 c0 81 01 0d c0 21 55 0d c1
  0x70:21 55 11 ce 81 01 00 00 00 00 00 00 00 00 00 11
   QSFP-DD Upper Page 00h:
       0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
  0x80:18 4a 55 4e 49 50 45 52 2d 31 54 31 20 20 20 20
  0x90:20 20 1b c9 37 34 30 2d 31 33 31 31 36 39 20 20
  0xa0:20 20 20 20 30 31 31 54 31 54 5a 46 41 37 32 35
  0xb0:30 34 33 20 20 20 32 32 30 36 31 37 20 20 43 4d
  0xc0:55 49 41 58 47 42 41 41 e0 5f 00 07 00 00 00 00
  0xd0:00 00 fe 00 05 00 00 00 00 00 00 00 00 00 e7 31
  0xe0:54 31 5a 46 41 20 20 20 20 20 20 20 20 20 20 30
  0xf0:31 31 30 20 20 20 20 20 20 20 20 20 20 20 20 20
   QSFP-DD Upper Page 01h:
       0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
  0x80:3d 16 01 02 8a 00 00 00 00 00 78 ea 82 1e 64 04
  0x90:79 c1 50 00 00 00 9c 18 00 cc 77 c3 07 0f 06 0f
  0xa0:17 1d 0d b7 ff 80 28 79 14 00 00 00 00 00 00 00
  0xb0:01 01 01 01 01 01 01 01 01 01 01 01 01 01 00 00
  0xc0:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0xd0:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0d
  0xe0:ce 21 55 0d cf 21 55 0d c2 21 55 41 c2 41 11 0d
  0xf0:c4 21 55 41 c4 41 11 ff 00 00 00 00 00 00 00 7a
   QSFP-DD Upper Page 02h:
       0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
  0x80:4c 00 fa 00 49 00 fd 00 90 88 75 30 8c a0 77 24
  0x90:7f ff 80 01 7f ff 80 01 50 00 b0 00 4b 00 d8 00
  0xa0:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0xb0:27 10 00 9e 18 a5 00 fb 00 00 00 00 00 00 00 00
  0xc0:3d e8 00 0f 27 10 00 32 00 00 00 00 00 00 00 00
  0xd0:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0xe0:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0xf0:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 5b
   Upper Page 10h:
       0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
  0x80:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x90:00 50 50 50 50 50 50 50 50 ff 00 00 55 55 55 55
  0xa0:ff ff 00 00 00 00 00 00 00 00 33 33 33 33 00 00
  0xb0:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0xc0:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0xd0:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0xe0:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0xf0:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
   Upper Page 11h:
       0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
  0x80:11 11 11 11 00 00 00 00 00 00 00 00 00 00 00 00
  0x90:00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00
  0xa0:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0xb0:00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00
  0xc0:00 00 00 00 00 00 00 00 00 00 11 11 11 11 50 50
  0xd0:50 50 50 50 50 50 ff 00 00 00 00 00 00 ff ff 44
  0xe0:44 44 44 00 00 00 00 22 22 22 22 00 00 00 00 00
  0xf0:11 00 00 00 00 00 00 00 11 00 00 00 00 00 00 00
  Warning: Page 10h,11h maybe stale
  Warning: Please run test picd optics cmd eeprom_rescan
  <snipped>

----------------


Four.  Per developer checking, the i2cdump command can prove some hardware issue existing on 61.22 version.

----------------
show chassis pic fpc-slot 0 pic-slot 0

FPC slot 0, PIC slot 0 information:
  Type                             8X400GE-MR + 4X100GE-MR
  State                            Online   
  PIC version                   255.09
  Uptime                           35 days, 20 hours, 55 minutes, 48 seconds
PIC port information:
                         Fiber                    Xcvr vendor       Wave-                     Xcvr          JNPR     MSA
  Port Cable type        type  Xcvr vendor        part number       length                    Firmware      Rev      Version
  0    400G-ZR-M         SM    JUNIPER-1T1        740-131169        1528.77 nm - 1567.13 nm   61.22         REV 01 
 
 
sh-5.0# i2cdump -f -y 2 0x50 
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 18 41 04 0a 00 00 00 00 05 00 00 00 00 00 1c 00    ?A??....?.....?.
10: 81 0d 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ??..............
20: 00 00 00 00 03 00 00 3d 16 00 00 00 00 00 00 00    ....?..=?.......
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
40: 20 14 18 16 00 00 00 00 00 00 00 00 00 00 00 00     ???............
50: 00 00 00 00 00 02 11 3e 81 01 11 3f 81 01 0d 3e    .....??>???????>
60: 21 55 11 c5 81 01 11 c0 81 01 0d c0 21 55 0d c1    !U??????????!U??
70: 21 55 11 ce 81 01 00 00 00 00 00 00 00 00 00 ba    !U????.........?
80: 00 00 00 00 00 00 00 00 00 00 00 19 00 00 00 00    ...........?....
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
b0: 00 00 00 00 00 00 00 00 00 00 03 e8 ff ff cd 38    ..........??..?8
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
 
 
Byte 3 bit 1 to 3 represents the module state as 5  which is fault as per the above dump.
Byte 8 is saying DataPathFirmwareFault set and L-ModuleStateChanged flag to 1.


----------------


Final Solution:

After upgrading the firmware version on port et-0/0/0 to 61.30, this issue was solved.
 

Modification History

2024-04-30 : Article Created
2024-05-01 : Contents Modified

2025-03-13 : Made it public