Upgraded the software of the ACX7100-48L devices, which requires reboot of the device for the new version to take effect. Two out of six of the devices completed the reboot, and their uplinks, QSFP56-DD-400GBASE-DR4 were offline.
The optical diagnostic showed no output power, and no received power.
We requested remote hands to re-seat the modules, which restored operation of the device uplinks.
Feb 25 13:23:51 JTAC picd: PICD_MODULE_ALARM: xcvr-0/0/48 Module StateMachine failed
This KB points to the minimum logs to be collected in issue state for further analysis of such issue.
From logs we can see that the interface was stuck with "Module StateMachine failed ".
The QSFP is supported one & there are no other logs apart from "Module StateMachine failed" in messages. The picd logs got averwritten.
The best is to engage JTAC soon to collect info, based on observations.
But if that is not feasible, below are the basic info which needs to be collected at the event time.
We can engage Engineering based on these info, if needed later. (replace et-0/0/48 in below CLIs with affected interface)
++++++++++++++++++++++++++++++++++++++++++++++++++++++
Ensure to save the picd logs immediately (else it can get overwritten)
show interfaces diagnostics optics et-0/0/48
show chassis pic fpc-slot 0 pic-slot 0
start shell pfe network fpc0
show picd optics fpc_slot 0 pic_slot 0 port 48 cmd diagnostics
show picd optics fpc_slot 0 pic_slot 0 port 48 cmd dump_devdb_info
show picd optics fpc_slot 0 pic_slot 0 port 48 cmd dump_xi_info
show picd optics fpc_slot 0 pic_slot 0 port 48 cmd error
show picd optics fpc_slot 0 pic_slot 0 port 48 cmd fec_status
show picd optics fpc_slot 0 pic_slot 0 port 48 cmd identifier
show picd optics fpc_slot 0 pic_slot 0 port 48 cmd info
show picd link-history fpc 0 pic 0 port 48 chan 0
show picd xcvr summary
show picd xcvr fpc 0 pic 0 port 48 cmd diagnostics
show picd xcvr fpc 0 pic 0 port 48 cmd firmware-info
show picd xcvr fpc 0 pic 0 port 48 cmd identifier
show picd xcvr fpc 0 pic 0 port 48 cmd info
show picd optics qsfpdd fpc_slot 0 pic_slot 0 port 48 cmd advertised_applications
show picd optics qsfpdd fpc_slot 0 pic_slot 0 port 48 cmd configured_params
show picd optics qsfpdd fpc_slot 0 pic_slot 0 port 48 cmd lpbk_pwr_consumption
show picd optics qsfpdd fpc_slot 0 pic_slot 0 port 48 cmd prbs_modes
show picd optics qsfpdd fpc_slot 0 pic_slot 0 port 48 cmd prbs_patterns
show picd optics qsfpdd fpc_slot 0 pic_slot 0 port 48 cmd prbs_status
show picd optics qsfpdd fpc_slot 0 pic_slot 0 port 48 cmd rx_pow_threshold
show picd optics qsfpdd fpc_slot 0 pic_slot 0 port 48 cmd tx_power_information