Meaning of syslog message ZTCHIP_CMERROR_FLEXMEM_INT_REG_FLEXMEM_PKT_INVLD_ACCESS_ERR and how to address it
ZTCHIP_CMERROR_FLEXMEM_INT_REG_FLEXMEM_PKT_INVLD_ACCESS_ERR occurs in ZTCHIP due to packet memory corruption when FLEXMEM: Packet side accessed a partition not allocated to it. Below logs may be seen during the issue period:
Jun 17 18:52:55 mx960 : %PFE-3: fpc3 Cmerror Op Set: XQSS(0): XQ-SS[0]: Packet-Reader fast eop request was made to an empty queue (URI: /fpc/3/pfe/0/cm/0/XQSS(0)/0/XQSS_CMERROR_PR_INT_REG_FEOP_DRY)
Jun 17 18:52:55 mx960 : %PFE-3: fpc3 Cmerror Op Set: XQSS(0): XQ-SS[0]: CPQW Fast request is asserted for empty Queue (URI: /fpc/3/pfe/0/cm/0/XQSS(0)/0/XQSS_CMERROR_CPQW_ERR_INT_FSET_FAST_DEQ_DRY_ERR)
Jun 17 18:53:02 mx960 : %PFE-3: fpc3 MQSS(0): DRD: UNROLL0: MC chunk address error in stage 4 - Chunk Address: 0x3f99a9d
Jun 17 18:53:28 mx960 : %PFE-3: fpc3 Cmerror Op Set: MQSS(0): MQSS(0): WO: Packet Error - Error Packets 255, Connection 33 (URI: /fpc/3/pfe/0/cm/0/MQSS(0)/0/ZTCHIP_MQSS_CMERROR_WO_INT_REG_PKT_ERR)
Jun 17 18:53:31 mx960 : %PFE-5: fpc3 CMError: /fpc/3/pfe/0/cm/0/MQSS(0)/0/ZTCHIP_MQSS_CMERROR_DRD_RORD_ENG_SRAM_PAR_PROTECT_FSET_REG_DETECTED_PQPTR (0x227f79), scope: pfe, category: functional, severity: minor, module: MQSS(0), type: DRD_RORD: Detected: PQ pointer memory, oc_category: default
/fpc/3/pfe/0/cm/0/MQSS(0)/0/ZTCHIP_MQSS_CMERROR_DRD_RORD_ENG_SRAM_PAR_PROTECT_FSET_REG_DETECTED_PQPTR (0x227f79) in module: MQSS(0) with scope: pfe category: functional level: minor, oc_category: default
Jun 17 18:53:31 mx960 : %PFE-3: fpc3 Cmerror Op Set: MQSS(0): MQSS(0): DRD: RORD0 Protect: Parity error detected for PQ pointer memory - data32_log_err 0x1, data32_log_address 0x42a (URI: /fpc/3/pfe/0/cm/0/MQSS(0)/0/ZTCHIP_MQSS_CMERROR_DRD_RORD_ENG_SRAM_PAR_PROTECT_FSET_REG_DETECTED_PQPTR)
/fpc/3/pfe/0/cm/0/MQSS(0)/0/ZTCHIP_MQSS_CMERROR_WO_INT_REG_PKT_ERR)
Jun 17 18:53:36 mx960 : %PFE-3: fpc3 Cmerror Op Set: MQSS(0): MQSS(0): WO: Packet Error - Error Packets 255, Connection 33 (URI: /fpc/3/pfe/0/cm/0/MQSS(0)/0/ZTCHIP_MQSS_CMERROR_WO_INT_REG_PKT_ERR)
Jun 17 18:53:47 mx960 : %PFE-3: fpc3 user.err ztchip-luss: dispatch_event_handler(1077): ZT[0:0].slice[0].disp[0] LMEM_PARCEL_MAX.
Jun 17 18:54:04 mx960 : %PFE-3: fpc3 Cmerror Op Set: ZT[0:0]: ZT[0:0]: FlexMem-9: Packet side accessed a partition not allocated to it (URI: /fpc/3/pfe/0/cm/0/ZT[0:0]/0/ZTCHIP_CMERROR_FLEXMEM_INT_REG_FLEXMEM_PKT_INVLD_ACCESS_ERR)
Jun 17 18:54:04 mx960 : %PFE-3: fpc3 Cmerror Op Set: MQSS(0): MQSS(0): WO: Packet Error - Error Packets 255, Connection 33 (URI: /fpc/3/pfe/0/cm/0/MQSS(0)/0/ZTCHIP_MQSS_CMERROR_WO_INT_REG_PKT_ERR)
*****Cmerror-counters*****
Identifier : /fpc/3/pfe/0/cm/0/ZT[0:0]/0/ZTCHIP_CMERROR_FLEXMEM_INT_REG_FLEXMEM_PKT_INVLD_ACCESS_ERR
Description : FLEXMEM: Packet side accessed a partition not allocated to it
Fru Type : fpc
Fru Slot : 3
Help String : No help info provided
Count : 4
Last Updated : 1718752174487
Error Module Id : 11
Component Id : 0
Error Code : 3538947
Occur Count : 4
Cleared Count : 0
Last Clear : 0
Action Count : 9
Last Action : 1718752174487
FMPC3(mx960_vty)# sh cmerror module 11 error 3538947
Error-id : 0x360003
Error Name : ZTCHIP_CMERROR_FLEXMEM_INT_REG_FLEXMEM_PKT_INVLD_ACCESS_ERR
State : enabled
Scope : pfe
Category : functional
PFE : 0
Configured Level : Major
Default Level : Major
Threshold : 1
Error Limit : 0
Clear Count : 0
OverItemThres Occur Count : 4
OverItemThres Clear Count : 0
Last-occurred(ms ago) : 440078243
Logs:
----------------------------------------------------------
Index Time Sub-Err State Description
0 06/18/24 23:09:34 0 Set unknown
1 06/18/24 23:09:34 0 Set unknown
2 06/18/24 23:09:34 0 Set unknown
3 06/18/24 23:09:34 0 Set unknown
Indications:
FlexMem is the on-chip memory - used for queuing data structures and on-chip delay bandwidth storage.
When the ZTCHIP receives traffic, if the incoming packet size is > 224B, the packet will be split into HEAD and TAIL with the HEAD size as 192B. The TAIL part of the packet will be stored in either the on-chip SRAM (FlexMem)* or off-chip HBM (High Bandwidth Memory)* by WI block [* Depending on the memory configuration, default is Hybrid where both FlexMem & HBM are used]. Once the packet becomes eligible to be sent out of PFE, the content will be read from FlexMem or HBM and the packet will be sent out via the Fabric/egress interface [depending on the packet look-up result]. The error code "ZTCHIP_CMERROR_FLEXMEM_INT_REG_FLEXMEM_PKT_INVLD_ACCESS_ERR" is seen occasionally, when the TAIL is stored in FlexMem and during its retrieval an invalid access error is observed. The error indicates an issue with accessing the FlexMem packet due to packet memory corruption, which could lead to packet processing problems or other issues.
This is a transient error. Depending on the preceding and superseding logs [if not self-corrected], it could lead to a wedge condition where the affected PFE will be disabled resulting in service impact.
Perform these steps to determine the cause and resolve the problem (if any). Continue through each step until the issue is resolved.
show log messages
show log chassisd
start shell network pfe <fpc#.0>
show nvram
show syslog messages
show cmerror modules brief ---> check for ZT[0:0]
---> check for
show cmerror modules <ZT[0:0] id > error <error id>
For instance:
FMPC3(mx960_vty)# sh cmerror module brief
-------------------------------------------------------------------------
Module Name Active Errors PFE Callback ModuleData
Specific Function
11 ZT[0:0] 0 Yes 0x00000000 0x190770720
show cmerror module 11 error <id> ---> replace id with error id from the logs
Tip: When looking at an event in the logs, it is important to focus on the first error message in a collection of syslog messages. The first error message is usually the cause of all the follow-on error messages. The follow-on collateral damage error messages can be ignored.