Syslog message ZTCHIP_MQSS_CMERROR_WO_INT_REG_PKT_ERR

Chassis Manager error "ZTCHIP_MQSS_CMERROR_WO_INT_REG_PKT_ERR" is reported when MQSS block in ZTCHIP encounters WO interrupt handler for packet error. Here are snippets of the logs seen during the issue period for reference:
Jul 25 00:33:48.000  bng-jtac-mx960-r2035-re0 fpc7 user.err ztchip-xqss: xqss_drop_poll_status:2261 XQ-SS[1]: DSTAT drop0 Qdepth underrun onqueue 136
Jul 25 00:33:48.842  bng-jtac-mx960-r2035-re0 inetd[14286]: Number of tftp connections at max limit (1)
Jul 25 00:33:48.853  bng-jtac-mx960-r2035-re0 fpc7 CMError: /fpc/7/pfe/0/cm/0/XQSS(1)/1/XQSS_CMERROR_DSTAT_INT_REG_DROP0_QDEPTH_UNDRN (0x230067), scope: pfe, category: functional, severity: major, module: XQSS(1), type: XQSS DSTAT DROP0 QDEPTH underrun, oc_category: unknown oc_category
Jul 25 00:33:48.853  bng-jtac-mx960-r2035-re0 fpc7 Performing action log for error /fpc/7/pfe/0/cm/0/XQSS(1)/1/XQSS_CMERROR_DSTAT_INT_REG_DROP0_QDEPTH_UNDRN (0x230067) in module: XQSS(1) with scope: pfe category: functional level: major, oc_category: unknown oc_category
Jul 25 00:33:48.853  bng-jtac-mx960-r2035-re0 fpc7 Performing action get-state for error /fpc/7/pfe/0/cm/0/XQSS(1)/1/XQSS_CMERROR_DSTAT_INT_REG_DROP0_QDEPTH_UNDRN (0x230067) in module: XQSS(1) with scope: pfe category: functional level: major, oc_category: unknown oc_category
Jul 25 00:33:48.943  bng-jtac-mx960-r2035-re0 inetd[14286]: Number of tftp connections at max limit (1)
Jul 25 00:33:49.318  bng-jtac-mx960-r2035-re0 chassisd[14276]: CHASSISD_FPC_ASIC_ERROR: <FPC 7> ASIC Error detected errorno 0x00230067 (null)
Jul 25 00:33:49.572  bng-jtac-mx960-r2035-re0 fpc7 MQSS(1): WO: COREIF dequeue watermark hit
Jul 25 00:33:49.572  bng-jtac-mx960-r2035-re0 fpc7 MQSS(1): WO: Packet Error - Error Packets 181, Connection 10
Jul 25 00:33:49.572  bng-jtac-mx960-r2035-re0 fpc7 ZT[1:0]: MCIF Rx: Checksum error detected on WO response - Chunk Address 0xc8b217
Jul 25 00:33:51.536  bng-jtac-mx960-r2035-re0 fpc7 MQSS(1): WO: Packet Error - Error Packets 255, Connection 10
Jul 25 00:33:51.536  bng-jtac-mx960-r2035-re0 fpc7 ZT[1:0]: MCIF Rx: Checksum error detected on WO response - Chunk Address 0x10b997b
Jul 25 00:33:53.541  bng-jtac-mx960-r2035-re0 fpc7 MQSS(1): WO: Packet Error - Error Packets 251, Connection 23
Jul 25 00:34:15.843  bng-jtac-mx960-r2035-re0 fpc7 CMError: /fpc/7/pfe/0/cm/0/XQSS(1)/1/XQSS_CMERROR_CPQW_ERR_INT_FSET_FAST_DEQ_DRY_ERR (0x2300e1), scope: pfe, category: functional, severity: major, module: XQSS(1), type: XQSS CPQW FAST DEQ DRY, oc_category: unknown oc_category
Jul 25 00:34:15.843  bng-jtac-mx960-r2035-re0 fpc7 Performing action log for error /fpc/7/pfe/0/cm/0/XQSS(1)/1/XQSS_CMERROR_CPQW_ERR_INT_FSET_FAST_DEQ_DRY_ERR (0x2300e1) in module: XQSS(1) with scope: pfe category: functional level: major, oc_category: unknown oc_category
Jul 25 00:34:15.843  bng-jtac-mx960-r2035-re0 fpc7 Performing action get-state for error /fpc/7/pfe/0/cm/0/XQSS(1)/1/XQSS_CMERROR_CPQW_ERR_INT_FSET_FAST_DEQ_DRY_ERR (0x2300e1) in module: XQSS(1) with scope: pfe category: functional level: major, oc_category: unknown oc_category
Jul 25 00:34:15.843  bng-jtac-mx960-r2035-re0 fpc7 Cmerror Op Set: XQSS(1): XQ-SS[1]: CPQW Fast request is asserted for empty Queue  (URI: /fpc/7/pfe/0/cm/0/XQSS(1)/1/XQSS_CMERROR_CPQW_ERR_INT_FSET_FAST_DEQ_DRY_ERR)
Jul 25 00:34:15.844  bng-jtac-mx960-r2035-re0 fpc7 CMError: /fpc/7/pfe/0/cm/0/MQSS(1)/1/ZTCHIP_MQSS_CMERROR_WO_INT_REG_PKT_ERR (0x22869c), scope: pfe, category: functional, severity: major, module: MQSS(1), type: WO: Packet Error, oc_category: default
Jul 25 00:34:15.844  bng-jtac-mx960-r2035-re0 fpc7 Performing action log for error /fpc/7/pfe/0/cm/0/MQSS(1)/1/ZTCHIP_MQSS_CMERROR_WO_INT_REG_PKT_ERR (0x22869c) in module: MQSS(1) with scope: pfe category: functional level: major, oc_category: default
Jul 25 00:34:15.844  bng-jtac-mx960-r2035-re0 fpc7 Performing action get-state for error /fpc/7/pfe/0/cm/0/MQSS(1)/1/ZTCHIP_MQSS_CMERROR_WO_INT_REG_PKT_ERR (0x22869c) in module: MQSS(1) with scope: pfe category: functional level: major, oc_category: default
Jul 25 00:34:15.844  bng-jtac-mx960-r2035-re0 fpc7 Cmerror Op Set: MQSS(1): MQSS(1): WO: Packet Error - Error Packets 255, Connection 31  (URI: /fpc/7/pfe/0/cm/0/MQSS(1)/1/ZTCHIP_MQSS_CMERROR_WO_INT_REG_PKT_ERR)

Jul 25 00:37:31.566  bng-jtac-mx960-r2035-re0 fpc7 Cmerror Op Set: MQSS(1): MQSS(1): WO: Packet Error - Error Packets 255, Connection 22  (URI: /fpc/7/pfe/0/cm/0/MQSS(1)/1/ZTCHIP_MQSS_CMERROR_WO_INT_REG_PKT_ERR)
Jul 25 00:37:33.568  bng-jtac-mx960-r2035-re0 fpc7 ZT[1:0]: MCIF Rx: Checksum error detected on WO response - Chunk Address 0x11399e1
Jul 25 00:37:33.568  bng-jtac-mx960-r2035-re0 fpc7 Cmerror Op Set: MQSS(1): MQSS(1): WO: Packet Error - Error Packets 255, Connection 32  (URI: /fpc/7/pfe/0/cm/0/MQSS(1)/1/ZTCHIP_MQSS_CMERROR_WO_INT_REG_PKT_ERR)
Jul 25 00:37:33.769  bng-jtac-mx960-r2035-re0 fpc7 Cmerror Op Set: MQSS(1): MQSS(1): DSTAT: DBGMON_CLIENT_USEMETER: Unexpected values - drop0_usemeter 15, drop1_usemeter 15, malloc_usemeter0 15, malloc_usemeter1 15, cpq_usemeter0 0, cpq_usemeter1 0  (URI: /fpc/7/pfe/0/cm/0/MQSS(1)/1/MQSS_CMERROR_DSTAT_DBGMON_CLIENT_USEMETER_UNEXPECTED)
Jul 25 00:37:34.235  bng-jtac-mx960-r2035-re0 chassisd[14276]: CHASSISD_FPC_ASIC_ERROR: <FPC 7> ASIC Error detected errorno 0x00220407 (null)
Jul 25 00:37:34.247  bng-jtac-mx960-r2035-re0 alarmd[15286]: Alarm set: FPC id=167774056, color=YELLOW, class=CHASSIS, reason=FPC 7 Minor Errors
Jul 25 00:37:34.247  bng-jtac-mx960-r2035-re0 craftd[14280]:  Minor alarm set, FPC 7 Minor Errors
Jul 25 00:37:36.000  bng-jtac-mx960-r2035-re0 fpc7 user.err packetio: [Error] Wedge-Detect : Host Loopback Wedge Detected: PFE: 1
Jul 25 00:37:36.000  bng-jtac-mx960-r2035-re0 fpc7 user.err packetio: [Error] Wedge-Detect : Host Loopback Wedge error block:  error counter: Host
Jul 25 00:37:36.592  bng-jtac-mx960-r2035-re0 fpc7 Cmerror Op Set: HstLbk:pfe:1: Host loopback timeout for pfe: 1 (URI: /fpc/7/pfe/0/cm/0/HstLbk:pfe:1/1/CMERROR_HOST_LOOPBACK_DECLARE_WEDGE)

 Indications:

• Usually, this error is a chain reaction to an original/underlying issue, so always check the preceding logs
• Ideally, this error will not cause any traffic impact or permanent PFE forwarding impact if not persistently seen as it is internally corrected
• A log will be registered to notify the condition 

This is a transient issue and depending on the frequency of the error occurrence could be impacting as the error could lead to a wedge condition blackholing traffic.
The "MQSS(*): WO: Packet Error - Error Packets 255" error is logged when a packet error is detected within the chip which compromises the integrity of the said packet. The packet error may be due to, but not limited to checksum error. This is a non-fatal error and it will cause said packet to be dropped.
If the packet errors are observed more than 255 times within 3 seconds, an error "ZTCHIP_MQSS_CMERROR_WO_INT_REG_PKT_ERR" will be logged. If seen more frequently/consistently it could also lead to a Wedge condition leading to a traffic black hole. 

Perform these steps to determine the cause and resolve the problem (if any). Continue through each step until the problem is resolved.

• Collect the show command output

Capture the output to a file (in case you have to open a technical support case). To do this, configure each SSH client/terminal emulator to log your session.
show log messages
show log chassisd
start shell network pfe <fpc#.0>
show nvram
show syslog messages
show cmerror modules brief ---> check for MQSS id [In current example error is in "MQSS(1)"] 
 
For instance:
FMPC7(bng-jtac-mx960-r2035-re0 vty)# sh cmerror module brief  
-------------------------------------------------------------------------
Module Name       Active Errors PFE   Callback  ModuleData
                     Specific Function       
-------------------------------------------------------------------------
17    MQSS(1)     0       Yes   0x149140800 0x3750258664
show cmerror module 17 error  <id> ---> replace id with error id [for instance 0x22869c from above logs]
 

• Analyze the show command output.
• In the 'show log messages', review the events that occurred at or just before the appearance of the "Fast request is asserted for empty Queue" message. Frequently, these events help identify the cause.
  • No RMA is required
  • Collect debugging information using  KB31922 - shell script collecting data if packet forwarding at Packet Forwarding Engine is compromised  [juniper.net]
  • If error is seen repeatedly, an FPC restart  during a maintenance window should clear this error

 
Tip: When looking at an event in the logs, it is important to focus on the first error message in a collection of syslog messages. The first error message is usually the cause of all the follow-on error messages. The follow-on collateral damage error messages can be ignored.