This article explains the syslog message ZTCHIP_MQSS_CMERROR_DRD_RORD_ENG_INT_REG_CMD_FSM_STATE_ERR (0x227fa3)
The "CMD FSM state error" message reports that the Forwarding State Machine of the reorder queue received an unexpected value. This indicates the lookup microcode within LUSS is sending an incorrect command parcel to the MQSS Center Chip. This seems a transient issue. The major alarms was raised and as an action PFE 0 got disabled.
Logs:-
Jun 13 20:22:23.000 2024 DEVICE fpc3 user.err ztchip-luss: ppe_error_interrupt(6365): ZT[0:0].slice[0]_PPE 25 Errors sync xtxn error
Jun 13 20:22:23.000 2024 DEVICE fpc3 user.err ztchip-luss: ppe_read_umem(1184): ZT[0:0].slice[0]_PPE 25 Invalid ppe_index 0.
Jun 13 20:22:23.000 2024 DEVICE fpc3 user.err ztchip-luss: create_ttrace_client(1594): error in ppe_cfg_read_and_extract_ucode_instr()
Jun 13 20:22:25.142 2024 DEVICE fpc3 ztchip_mcif_tx1_intr_handler(16679): ZT[0:0]: MCIF Tx1: Detected out-of-range request -data64_err_log 0xa829c0fd2f4
Jun 13 20:22:25.142 2024 DEVICE inetd[32254]: Number of tftp connections at max limit (1)
Jun 13 20:22:25.149 2024 DEVICE fpc3 MQSS(0): DRD: Error: WAN reorder ID timeout error - Valid 1, Reorder ID 246
Jun 13 20:22:25.149 2024 DEVICE fpc3 MQSS(0): DRD: RORD0: CMD reorder ID error - Command 1, Reorder ID 246, QID 16384
Jun 13 20:22:25.150 2024 DEVICE fpc3 Error: /fpc/3/pfe/0/cm/0/MQSS(0)/0/ZTCHIP_MQSS_CMERROR_DRD_RORD_ENG_INT_REG_CMD_FSM_STATE_ERR (0x227fa3), scope: pfe, category: functional, severity: major, module: MQSS(0), type: DRD_RORD_ENG_INT: CMD FSM State Error
Jun 13 20:22:25.150 2024 DEVICE fpc3 Performing action get-state for error /fpc/3/pfe/0/cm/0/MQSS(0)/0/ZTCHIP_MQSS_CMERROR_DRD_RORD_ENG_INT_REG_CMD_FSM_STATE_ERR (0x227fa3) in module: MQSS(0) with scope: pfe category: functional level: major
Jun 13 20:22:45.000 2024 DEVICE fpc3 user.notice root: pfe_get_state.sh : Running PFE get state script
Jun 13 20:22:45.000 2024 DEVICE fpc3 user.notice root: pfe_get_state.sh : PFE AFT Get state started - Module: (31), Error: (2260899), Date/Time: (13/06/2024 20:22:45)
Jun 13 20:22:53.747 2024 DEVICE fpc3 platform_zt_module_get_state_ack_cb: module id:31 action 2 error-code 227fa3 ack 0 timed out
Jun 13 20:22:53.749 2024 DEVICE fpc3 Performing action cmalarm for error /fpc/3/pfe/0/cm/0/MQSS(0)/0/ZTCHIP_MQSS_CMERROR_DRD_RORD_ENG_INT_REG_CMD_FSM_STATE_ERR (0x227fa3) in module: MQSS(0) with scope: pfe category: functional level: major
Jun 13 20:22:53.749 2024 DEVICE fpc3 Performing action disable-pfe for error /fpc/3/pfe/0/cm/0/MQSS(0)/0/ZTCHIP_MQSS_CMERROR_DRD_RORD_ENG_INT_REG_CMD_FSM_STATE_ERR (0x227fa3) in module: MQSS(0) with scope: pfe category: functional level: major
Jun 13 20:22:53.749 2024 DEVICE fpc3 PFE 0: 'PFE Disable' action performed. Bringing down ifd et-3/0/0 288
Jun 13 20:22:53.749 2024 DEVICE fpc3 PFE 0: 'PFE Disable' action performed. Bringing down ifd et-3/0/1 289
Jun 13 20:22:53.801 2024 DEVICE fpc3 PFE 0: 'PFE Disable' action performed. Bringing down ifd et-3/0/2 290
Jun 13 20:22:53.858 2024 DEVICE fpc3 PFE 0: 'PFE Disable' action performed. Bringing down ifd et-3/0/3 291
Jun 13 20:22:53.931 2024 DEVICE fpc3 Cmerror Op Set: MQSS(0): MQSS(0): DRD: RORD0: CMD FSM state error - Command 13, Reorder ID 246, QID 5074, FSM State 7 (URI: /fpc/3/pfe/0/cm/0/MQSS(0)/0/ZTCHIP_MQSS_CMERROR_DRD_RORD_ENG_INT_REG_CMD_FSM_STATE_ERR)
Jun 13 20:22:54.000 2024 DEVICE fpc3 user.err aftd-trio: [Error] PFE:Invalid pfeInst, pfeInst:
Perform these steps to determine the cause and resolve the problem (if any). Continue through each step until the problem is resolved.
Capture the output to a file (in case you have to open a technical support case). To do this, configure each SSH client/terminal emulator to log your session.
show log messages
show log chassisd
start shell network pfe <fpc#>
show nvram
show syslog messages
exit
2.Analyze the show command output.
In the 'show log messages', review the events that occurred at or just before the appearance of the "CMD FSM state error" message. Frequently these events help identify the cause.