Article Detail

Description

This article explains the meaning of the syslog error DSTAT_DBGMON_CLIENT_USEMETER syslog and how to address it

Symptoms

Every PFE has a limited buffer resource to hold packets. When you have a high volume of traffic passing through a PFE and congestion happens, said buffer will be used to hold the packets. In cases where PFE is continuously congested due to high utilization, it could potentially block the PFE from serving the new incoming packets and thereby packet drops. The current buffer utilization is measured using usemeter. Thus, when the usemeter value reaches the threshold, it may indicate that the packet memory buffers are not released on the pfe leading to free memory exhaustion.

Here is a snippet of the error encountered on MPC10E card for reference:

May 17 19:34:02 bng-jtac-mx960-r2035-re0 : %PFE-5: fpc3 Performing action log for error /fpc/3/pfe/0/cm/0/XQSS(0)/0/XQSS_CMERROR_DSTAT_INT_REG_DROP0_QDEPTH_UNDRN (0x230067) in module: XQSS(0) with scope: pfe category: functional level: major, oc_category: unknown oc_category

May 17 19:34:02 bng-jtac-mx960-r2035-re0 : %PFE-5: fpc3 Performing action get-state for error /fpc/3/pfe/0/cm/0/XQSS(0)/0/XQSS_CMERROR_DSTAT_INT_REG_DROP0_QDEPTH_UNDRN (0x230067) in module: XQSS(0) with scope: pfe category: functional level: major, oc_category: unknown oc_category

May 17 19:34:02 bng-jtac-mx960-r2035-re0 inetd[13645]: %DAEMON-4: Number of tftp connections at max limit (1)

May 17 19:34:02 bng-jtac-mx960-r2035-re0 chassisd[13638]: %DAEMON-3-CHASSISD_FPC_ASIC_ERROR: <FPC 3> ASIC Error detected errorno 0x00230067 (null)

May 17 19:34:03 bng-jtac-mx960-r2035-re0 : %PFE-3: fpc3 MQSS(0): WO: COREIF dequeue watermark hit

May 17 19:34:03 bng-jtac-mx960-r2035-re0 : %PFE-3: fpc3 MQSS(0): WO: Packet Error - Error Packets 231, Connection 10

May 17 19:34:03 bng-jtac-mx960-r2035-re0 : %PFE-3: fpc3 ZT[0:0]: MCIF Rx: Checksum error detected on WO response - Chunk Address 0x53beb2

May 17 19:34:11 bng-jtac-mx960-r2035-re0 : %PFE-3: fpc3 MQSS(0): WO: Packet Error - Error Packets 255, Connection 10

May 17 19:34:11 bng-jtac-mx960-r2035-re0 : %PFE-3: fpc3 ZT[0:0]: MCIF Rx: Checksum error detected on WO response - Chunk Address 0xa1ff43

May 17 19:34:21 bng-jtac-mx960-r2035-re0 : %PFE-5: fpc3 user.notice root: pfe_get_state.sh : Running PFE get state script

May 17 19:34:21 bng-jtac-mx960-r2035-re0 : %PFE-5: fpc3 user.notice root: pfe_get_state.sh : PFE AFT Get state started - Module: (27), Error: (2293863), Date/Time: (17/05/2024 19:34:21)

May 17 19:34:21 bng-jtac-mx960-r2035-re0 : %PFE-5: fpc3 user.notice root: pfe_get_state.sh : Executing default CLI commands

May 17 19:34:22 bng-jtac-mx960-r2035-re0 : %PFE-3: fpc3 user.err packetio: [Error] Wedge-Detect : Host Loopback Wedge Detected: PFE: 0

May 17 19:34:22 bng-jtac-mx960-r2035-re0 : %PFE-3: fpc3 user.err packetio: [Error] Wedge-Detect : Host Loopback Wedge error block: error counter: Host

May 17 19:34:29 bng-jtac-mx960-r2035-re0 : %PFE-5: fpc3 Performing action cmalarm for error /fpc/3/pfe/0/cm/0/XQSS(0)/0/XQSS_CMERROR_CPQW_ERR_INT_FSET_FAST_DEQ_DRY_ERR (0x2300e1) in module: XQSS(0) with scope: pfe category: functional level: major, oc_category: unknown oc_category

May 17 19:34:29 bng-jtac-mx960-r2035-re0 : %PFE-3: fpc3 Cmerror Op Set: XQSS(0): XQ-SS[0]: CPQW Fast request is asserted for empty Queue (URI: /fpc/3/pfe/0/cm/0/XQSS(0)/0/XQSS_CMERROR_CPQW_ERR_INT_FSET_FAST_DEQ_DRY_ERR)

May 17 19:34:29 bng-jtac-mx960-r2035-re0 : %PFE-5: fpc3 CMError: /fpc/3/pfe/0/cm/0/MQSS(0)/0/MQSS_CMERROR_DSTAT_DBGMON_CLIENT_USEMETER_UNEXPECTED (0x220407), scope: pfe, category: functional, severity: minor, module: MQSS(0), type: DSTAT: DBGMON_CLIENT_USEMETER: Unexpected usemeter values, oc_category: default ---> suspected this to be due to PR1772826, however, the conditions don't match

May 17 19:34:29 bng-jtac-mx960-r2035-re0 : %PFE-5: fpc3 Performing action log for error /fpc/3/pfe/0/cm/0/MQSS(0)/0/MQSS_CMERROR_DSTAT_DBGMON_CLIENT_USEMETER_UNEXPECTED (0x220407) in module: MQSS(0) with scope: pfe category: functional level: minor, oc_category: default

May 17 19:34:29 bng-jtac-mx960-r2035-re0 : %PFE-5: fpc3 Performing action cmalarm for error /fpc/3/pfe/0/cm/0/MQSS(0)/0/MQSS_CMERROR_DSTAT_DBGMON_CLIENT_USEMETER_UNEXPECTED (0x220407) in module: MQSS(0) with scope: pfe category: functional level: minor, oc_category: default

May 17 19:34:29 bng-jtac-mx960-r2035-re0 : %PFE-3: fpc3 Cmerror Op Set: MQSS(0): MQSS(0): DSTAT: DBGMON_CLIENT_USEMETER: Unexpected values - drop0_usemeter 15, drop1_usemeter 15, malloc_usemeter0 15, malloc_usemeter1 15, cpq_usemeter0 1, cpq_usemeter1 0 (URI: /fpc/3/pfe/0/cm/0/MQSS(0)/0/MQSS_CMERROR_DSTAT_DBGMON_CLIENT_USEMETER_UNEXPECTED)

May 17 19:34:30 bng-jtac-mx960-r2035-re0 : %PFE-3: fpc3 Cmerror Op Set: MQSS(0): MQSS(0): DSTAT: DBGMON_CLIENT_USEMETER: Unexpected values - drop0_usemeter 15, drop1_usemeter 15, malloc_usemeter0 15, malloc_usemeter1 15, cpq_usemeter0 0, cpq_usemeter1 0 (URI: /fpc/3/pfe/0/cm/0/MQSS(0)/0/MQSS_CMERROR_DSTAT_DBGMON_CLIENT_USEMETER_UNEXPECTED)

May 17 19:34:31 bng-jtac-mx960-r2035-re0 rpd[14771]: %DAEMON-5-RPD_LDP_NBRDOWN: LDP neighbor 189.246.213.194 (et-3/0/4.0) is down

May 17 19:34:31 bng-jtac-mx960-r2035-re0 : %PFE-3: fpc3 Cmerror Op Set: MQSS(0): MQSS(0): DSTAT: DBGMON_CLIENT_USEMETER: Unexpected values - drop0_usemeter 15, drop1_usemeter 15, malloc_usemeter0 15, malloc_usemeter1 15, cpq_usemeter0 0, cpq_usemeter1 0 (URI: /fpc/3/pfe/0/cm/0/MQSS(0)/0/MQSS_CMERROR_DSTAT_DBGMON_CLIENT_USEMETER_UNEXPECTED)

May 17 19:34:32 bng-jtac-mx960-r2035-re0 chassisd[13638]: %DAEMON-3-CHASSISD_FPC_ASIC_ERROR: <FPC 3> ASIC Error detected errorno 0x00230063 (null)

May 17 19:34:32 bng-jtac-mx960-r2035-re0 alarmd[14805]: %DAEMON-4: Alarm set: FPC id=150995816, color=RED, class=CHASSIS, reason=FPC 3 Major Errors

May 17 19:34:32 bng-jtac-mx960-r2035-re0 craftd[13641]: %DAEMON-4: Major alarm set, FPC 3 Major Errors

May 17 19:34:32 bng-jtac-mx960-r2035-re0 alarmd[14805]: %DAEMON-4: Alarm set: FPC id=167773032, color=YELLOW, class=CHASSIS, reason=FPC 3 Minor Errors

May 17 19:34:37 bng-jtac-mx960-r2035-re0 lfmd[13660]: %DAEMON-3: Action Syslog on et-3/0/4 [OAM_UNINET_SESSION]: :Adjacency Lost

The above error suggests packets are stuck and accumulated in mqss for a given pfe and are not transmitted out of mqss. This error is a chain reaction to events that led to packet accumulation and thereby memory exhaustion. You may also see other events reported with this at times like pfe getting wedged or self-ping failure.

*Note - Depending on the FPC model the chip name in the syslog and the show commands at the pfe shell may differ.

Solution

1. In the 'show log messages', review the events that occurred at or just before the appearance of the MQSS_CMERROR_DSTAT_DBGMON_CLIENT_USEMETER_UNEXPECTED error message. Frequently these events help in identifying the root cause.

2.Track the usemeter via show mqss 0 dstat usemeter [FPC shell cmd]

For instance:

Start shell pfe network fpc3

SMPC3(bng-jtac-mx960-r2035-re0 vty)# show mqss x dstat usemeter [replace x with pfe id where the congestion is seen. From logs in Symptoms section we see the error was logged for FPC 3 PFE 0. Note that depending on the FPC model the chip name in the syslog and the show commands at the pfe shell may differ]

DSTAT Usemeter Information

--------------------------

--------------------------------------------------------

Scheduler DSTAT Usemeter MALLOC Usemeter CPQ Usemeter

--------------------------------------------------------

WAN 0 0 0

Fabric 0 0 0

--------------------------------------------------------

3. Check out the below PRs and verify it is applicable:

https://prsearch.juniper.net/PR1767970

https://prsearch.juniper.net/PR1772826

4. If Point 3 is not applicable, please open a case with JTAC for analysis

Modification History

2024-05-23 : Article Created