The meaning of the error "MQSS_CMERROR_WO_INT_REG_PACK_ERR (0x220af6)" and the steps to address it

The error "MQSS_CMERROR_WO_INT_REG_PACK_ERR (0x220af6)" indicates that a packet is corrupted at the MQSS block, causing checksum errors when the packet ingress the EACHIP. Due to this error congestion was observed on FPC 2 PFE 0 as other FPCs were unable to reach PFE 0 and it was disabled to avoid traffic blackholing.

A snippet of the logs for reference:
root@mx10008-re0> show chassis alarms no-forwarding
1 alarms currently active
Alarm time    Class Description
2023-10-20 09:13:40 UTC Major FPC 2 Major Errors

root@mx10008-re0> show chassis hardware detail no-forwarding
Hardware inventory:
Item    Version Part number Serial number  Description
Chassis        EX828    JNP10008 [MX10008]
FPC 2   REV 23 750-084779 BCCD1085   JNP10K-LC2101
 CPU   REV 09 750-073391 BCCD5302   LC 2101 PMB
 PIC 0     BUILTIN  BUILTIN   4xQSFP28 SYNCE

root@mx10008-re0> show chassis fabric fpcs no-forwarding
Fabric management FPC state:
FPC 2
 PFE #0
    : Fabric Disabled

root@mx10008-re0> show chassis fabric plane no-forwarding
Fabric management PLANE state
Plane 0
 Plane state: ACTIVE
  FPC 2
   PFE 0 :Fabric Disabled
Plane 1
 Plane state: ACTIVE
  FPC 2
   PFE 0 :Fabric Disabled
Plane 2
 Plane state: ACTIVE
  FPC 2
   PFE 0 :Fabric Disabled
Plane 3
 Plane state: ACTIVE
  FPC 2
   PFE 0 :Fabric Disabled
Plane 4
 Plane state: ACTIVE
  FPC 2
   PFE 0 :Fabric Disabled
Plane 5
 Plane state: ACTIVE
  FPC 2
   PFE 0 :Fabric Disabled

Log messages:
Oct 20 09:12:58 mx10008-re0 fpc2 MQSS(0): FO: Packet Error - Error Packets 2, Stream 81, Port 3 
Oct 20 09:12:58 mx10008-re0 fpc2 MQSS(0): WO: Packet Error - Error Packets 1, Connection 25 
Oct 20 09:12:58 mx10008-re0 fpc2 eachip_hmcif_rx_intr_handler(7239): EA[0:0]: HMCIF Rx: Checksum error detected on WO response - Chunk Address 0xcec05b 
Oct 20 09:12:58 mx10008-re0 fpc2 eachip_hmcif_rx_intr_handler(7333): EA[0:0]: HMCIF Rx: Checksum error detected on FO response - Chunk Address 0x72ffea 
Oct 20 09:12:58 mx10008-re0 fpc3 MQSS(1): FI: Error cell sent to reorder engine - Stream 32, Count 1 
Oct 20 09:12:58 mx10008-re0 fpc5 MQSS(1): FI: Error cell sent to reorder engine - Stream 32, Count 1 
Oct 20 09:12:59 mx10008-re0 fpc2 MQSS(0): FO: Packet Error - Error Packets 255, Stream 64, Port 20 
Oct 20 09:12:59 mx10008-re0 inetd[17210]: Number of tftp connections at max limit (1)
Oct 20 09:12:59 mx10008-re0 fpc2 MQSS(0): FO: Output buffer deallocation failure 
Oct 20 09:12:59 mx10008-re0 fpc2 MQSS(0): FI: Error cell sent to reorder engine - Stream 32, Count 495 
Oct 20 09:12:59 mx10008-re0 fpc2 MQSS(0): WO: Packet Error - Error Packets 1, Connection 12 
Oct 20 09:12:59 mx10008-re0 fpc2 MQSS(0): PKTGRP: Missing SOP in transactions from DRD 
Oct 20 09:12:59 mx10008-re0 fpc2 MQSS(0): CPQF: Queue underrun indication - Queue 33 
Oct 20 09:12:59 mx10008-re0 fpc2 eachip_hmcif_tx_intr_handler(3531): EA[0:0]: HMCIF Tx: Out-of-range request detected - Client ID 0, Request Type 0, Request Size 3, Link Number 1, Request Address 0x5a33580 
Oct 20 09:12:59 mx10008-re0 fpc2 eachip_hmcif_rx_intr_handler(7239): EA[0:0]: HMCIF Rx: Checksum error detected on WO response - Chunk Address 0xe24592 
Oct 20 09:12:59 mx10008-re0 fpc2 eachip_hmcif_rx_intr_handler(7265): EA[0:0]: HMCIF Rx: Injected checksum error detected on WO response - Chunk Address 0x2073885 
Oct 20 09:12:59 mx10008-re0 fpc2 eachip_hmcif_rx_intr_handler(7333): EA[0:0]: HMCIF Rx: Checksum error detected on FO response - Chunk Address 0xbd4eab 
Oct 20 09:12:59 mx10008-re0 fpc2 eachip_hmcif_rx_intr_handler(7359): EA[0:0]: HMCIF Rx: Injected checksum error detected on FO response - Chunk Address 0xa804ac 
Oct 20 09:12:59 mx10008-re0 fpc2 MQSS(0): DRD: UNROLL0: Not enough chunks allocated by RTL for a packet 
Oct 20 09:12:59 mx10008-re0 fpc2 MQSS(0): DRD: UNROLL0: HMC chunk length error in stage 4 - Chunk Address: 0xe73574 
Oct 20 09:12:59 mx10008-re0 fpc2 MQSS(0): DRD: UNROLL0: HMC chunk address error in stage 4 - Chunk Address: 0x0 
Oct 20 09:12:59 mx10008-re0 fpc2 MQSS(0): DRD: UNROLL1: Not enough chunks allocated by RTL for a packet 
Oct 20 09:12:59 mx10008-re0 fpc2 MQSS(0): DRD: UNROLL1: HMC chunk length error in stage 4 - Chunk Address: 0x4de75b 
Oct 20 09:12:59 mx10008-re0 fpc2 MQSS(0): DRD: UNROLL1: HMC chunk address error in stage 4 - Chunk Address: 0x0 
Oct 20 09:12:59 mx10008-re0 fpc2 MQSS(5): FI: Error cell sent to reorder engine - Stream 32, Count 102 
Oct 20 09:12:59 mx10008-re0 fpc2 CMError: /fpc/2/pfe/0/cm/0/MQSS(0)/0/MQSS_CMERROR_WO_INT_REG_PACK_ERR (0x220af6), scope: pfe, category: functional, severity: major, module: MQSS(0), type: WO: Packing Error, oc_category: default 
Oct 20 09:12:59 mx10008-re0 fpc2 Performing action get-state for error /fpc/2/pfe/0/cm/0/MQSS(0)/0/MQSS_CMERROR_WO_INT_REG_PACK_ERR (0x220af6) in module: MQSS(0) with scope: pfe category: functional level: major, oc_
category: default 
Oct 20 09:12:59 mx10008-re0 fpc0 Cmerror Op Set: CM[5]: MPC fabric remote PFE error (rate based) (1) exceed raising threshold (1) occurrance (0) for module/pfe (13:5) (URI: /fpc/0/pfe/0/cm/0/CM[5]/5/CM_CMERROR_FABRIC
_REMOTE_PFE_RATE) 
Oct 20 09:12:59 mx10008-re0 fpc0 Cmerror Op Set: CM[5]: MPC fabric remote PFE error (aggregate based) (1) exceed raising threshold (1) occurrance (0) for module/pfe (13:5) (URI: /fpc/0/pfe/0/cm/0/CM[5]/5/CM_CMERROR_F
ABRIC_REMOTE_PFE_AGGR) 
Oct 20 09:12:59 mx10008-re0 fpc2 EA[0:0]-0 Congestion Detected, Active Zones f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:0:0:f:f:f:f 
Oct 20 09:13:00 mx10008-re0 fpc2 EA[0:0]-1 Congestion Detected, Active Zones:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f 
Oct 20 09:13:00 mx10008-re0 fpc2 EA[0:0]-2 Congestion Detected, Active Zones:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f 
Oct 20 09:13:00 mx10008-re0 fpc2 EA[0:0]-3 Congestion Detected, Active Zones:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f 
Oct 20 09:13:00 mx10008-re0 fpc3 Cmerror Op Set: CM[0]: MPC fabric remote PFE error (rate based) (1) exceed raising threshold (1) occurrance (0) for module/pfe (8:0) (URI: /fpc/3/pfe/0/cm/0/CM[0]/0/CM_CMERROR_FABRIC_
REMOTE_PFE_RATE) 
Oct 20 09:13:00 mx10008-re0 fpc3 Cmerror Op Set: CM[0]: MPC fabric remote PFE error (aggregate based) (1) exceed raising threshold (1) occurrance (0) for module/pfe (8:0) (URI: /fpc/3/pfe/0/cm/0/CM[0]/0/CM_CMERROR_FA
BRIC_REMOTE_PFE_AGGR) 
Oct 20 09:13:00 mx10008-re0 fpc7 MQSS(1): RDR: Detected error cells into engine - Stream 32, Count 4436 
Oct 20 09:13:00 mx10008-re0 fpc7 MQSS(1): RDR: Packet errors detected in PSV block - Stream 32, Count 22 
Oct 20 09:13:00 mx10008-re0 fpc7 MQSS(1): RDR: Packet transfer to ICM was errored - Stream 32, Count 4414 
Oct 20 09:13:01 mx10008-re0 fpc4 Cmerror Op Set: CM[2]: MPC fabric remote PFE error (rate based) (1) exceed raising threshold (1) occurrance (0) for module/pfe (10:2) (URI: /fpc/4/pfe/0/cm/0/CM[2]/2/CM_CMERROR_FABRIC
_REMOTE_PFE_RATE) 
Oct 20 09:13:01 mx10008-re0 fpc4 Cmerror Op Set: CM[2]: MPC fabric remote PFE error (aggregate based) (1) exceed raising threshold (1) occurrance (0) for module/pfe (10:2) (URI: /fpc/4/pfe/0/cm/0/CM[2]/2/CM_CMERROR_F
ABRIC_REMOTE_PFE_AGGR) 
Oct 20 09:13:01 mx10008-re0 fpc2 ppe_error_interrupt(5538): EA[0:0]_PPE 28 Errors thread timeout error 
Oct 20 09:13:01 mx10008-re0 fpc5 Cmerror Op Set: CM[2]: MPC fabric remote PFE error (rate based) (1) exceed raising threshold (1) occurrance (0) for module/pfe (10:2) (URI: /fpc/5/pfe/0/cm/0/CM[2]/2/CM_CMERROR_FABRIC
_REMOTE_PFE_RATE) 
Oct 20 09:13:01 mx10008-re0 fpc5 Cmerror Op Set: CM[2]: MPC fabric remote PFE error (aggregate based) (1) exceed raising threshold (1) occurrance (0) for module/pfe (10:2) (URI: /fpc/5/pfe/0/cm/0/CM[2]/2/CM_CMERROR_F
ABRIC_REMOTE_PFE_AGGR) 
Oct 20 09:13:01 mx10008-re0 fpc2 ppe_error_interrupt(5538): EA[0:0]_PPE 29 Errors thread timeout error 
Oct 20 09:13:01 mx10008-re0 fpc2 EA[0:0]-0 Congestion Detected, Active Zones f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:0:0:f:f:f:f 
Oct 20 09:13:01 mx10008-re0 fpc2 EA[0:0]-1 Congestion Detected, Active Zones:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f 
Oct 20 09:13:01 mx10008-re0 fpc2 EA[0:0]-2 Congestion Detected, Active Zones:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f 
Oct 20 09:13:01 mx10008-re0 fpc2 EA[0:0]-3 Congestion Detected, Active Zones:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f 
Oct 20 09:13:01 mx10008-re0 fpc2 ppe_step_all_ppe_context(9291): EA[0:0]_PPE 0 cntx_single_step_timeout busy 0x00000004 
Oct 20 09:13:01 mx10008-re0 fpc2 PFE_ERROR_FAIL_OPERATION: Wedge op single step failed  
Oct 20 09:13:40 mx10008-re0 fpc2 Performing action cmalarm for error /fpc/2/pfe/0/cm/0/MQSS(0)/0/MQSS_CMERROR_WO_INT_REG_PACK_ERR (0x220af6) in module: MQSS(0) with scope: pfe category: functional level: major, oc_category: default 
Oct 20 09:13:40 mx10008-re0 fpc2 Performing action disable-pfe for error /fpc/2/pfe/0/cm/0/MQSS(0)/0/MQSS_CMERROR_WO_INT_REG_PACK_ERR (0x220af6) in module: MQSS(0) with scope: pfe category: functional level: major, oc_category: default 
Oct 20 09:13:40 mx10008-re0 fpc2 PFE 0: 'PFE Disable' action performed. Bringing down ifd et-2/0/0 362 
Oct 20 09:13:40 mx10008-re0 fpc2 PFE 0: 'PFE Disable' action performed. Bringing down ifd et-2/0/1 363 
Oct 20 09:13:40 mx10008-re0 alarmd[18306]: Alarm set: FPC id=150995560, color=RED, class=CHASSIS, reason=FPC 2 Major Errors
Oct 20 09:13:40 mx10008-re0 craftd[17203]: Receive FX craftd set alarm message: color: 1 class: 100 object: 104 slot: 2 silent: 0 short_reason=FPC 2 Major Errors long_reason=FPC 2 Major Errors id=150995560 reason=150
994944
Oct 20 09:13:40 mx10008-re0 craftd[17203]: Major alarm set, FPC 2 Major Errors

Config Snippet:
set chassis fpc 2 apply-groups FPC_ALARMS_LOG
set groups FPC_ALARMS_LOG chassis fpc <*> error major action get-state
set groups FPC_ALARMS_LOG chassis fpc <*> error major action alarm
set groups FPC_ALARMS_LOG chassis fpc <*> error major action disable  ----> When set it would disable the pfe when the error is encountered

Below are logs indicating pfe-diable action due to the cmerror:
Oct 20 09:13:40 mx10008-re0 fpc2 Performing action disable-pfe for error /fpc/2/pfe/0/cm/0/MQSS(0)/0/MQSS_CMERROR_WO_INT_REG_PACK_ERR (0x220af6) in module: MQSS(0) with scope: pfe category: functional level: major, oc_category: default 
Oct 20 09:13:40 mx10008-re0 fpc2 PFE 0: 'PFE Disable' action performed. Bringing down ifd et-2/0/0 362 
Oct 20 09:13:40 mx10008-re0 fpc2 PFE 0: 'PFE Disable' action performed. Bringing down ifd et-2/0/1 363 
 

• Collect the following show command outputs to investigate further.

• Analyze the show command outputs matching the time when the alarm was triggered. Check where the error was reported on which block it was detected on the EACHIP, and if it was preceded by any incidents.

• Execute the below steps during MW and omit subsequent steps if the issue is resolved