Description

This article discusses a scenario when FPC Major Error HOST_LOOPBACK_MAKE_CMERROR_ID, "HOST LOOPBACK WEDGE DETECTED IN PATH ID <>" is reported and clarifies whether any actions need to be taken.

Symptoms

Device reports the following FPC Major Errors alarm, the PATH ID[x] and CMERROR ID ( ) may vary depends on which host stream get wedged on different card (in the example below FPC in slot 4 reported this alarm). Usually this error is reported for multiple PATH IDs.

 

  • show system alarms
user@device> show chassis alarms
1 alarms currently active
Alarm time Class Description
<date> <time> Major FPC 4 Major Errors
  • show chassis errors active detail fpc-slot 4
NGMPC4(router-re0 vty)# show cmerror module 4        

Module (4) (Host Loopback)
PFE support 0
Get state cb  0x10a9411c platform action cb 0x0 control cb 0x0

Error-id     PFE  Level  Threshold  Count  Occured  Cleared  Last-occurred(ms ago)  Name
------------------------------------------------------------------------------------------
0x020001      0   Major      1        1       1        0        164049              HOST_LOOPBACK_MAKE_CMERROR_ID[0]
0x020002      0   Major      1        0       0        0        0                   HOST_LOOPBACK_MAKE_CMERROR_ID[1]
0x020003      0   Major      1        0       0        0        0                   HOST_LOOPBACK_MAKE_CMERROR_ID[2]

NGMPC4(router-re0 vty)# show cmerror module 4  error 0x020001

Error-id              : 0x20001

Error Name                : HOST_LOOPBACK_MAKE_CMERROR_ID[0]
Identifier                : /fpc/4/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[0]
Description               : Host Loopback Path Id 0
State                     : enabled
Scope                     : pfe
Category                  : functional
PFE                       : 0
Configured Level          : Major
Default Level             : Major
Count                     : 1
Threshold                 : 1
Error Limit               : 0
Occur Count               : 1
Clear Count               : 0
OverItemThres Occur Count : 1
OverItemThres Clear Count : 0
Last-occurred(ms ago) : 173190
  • show log messages
router-re0 fpc4 CMError: /fpc/4/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[0] (0x20001), scope: pfe, category: functional, severity: major, module: Host Loopback, type: Host Loopback Path Id 0
router-re0 fpc4 Performing action get-state for error /fpc/4/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[0] (0x20001) in module: Host Loopback with scope: pfe category: functional level: major
router-re0 fpc4 Performing action cmalarm for error /fpc/4/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[0] (0x20001) in module: Host Loopback with scope: pfe category: functional level: major
router-re0 alarmd[26268]: Alarm set: FPC id=150995304, color=RED, class=CHASSIS, reason=FPC 4 Major Errors

Indications:

  • PFE wedge of the WAN forwarding block/circuit is detected and traffic forwarding is compromised
  • Major alarm will be raised
  • By default, disable-pfe action will be taken if Major Error alarm is raised.

Solution

The host-path from the WAN interfaces is always validated by periodic host loopback probes. If this forwarding path healthiness validation fails for five seconds, a major alarm is raised. In the majority of all events, this is a post effect of some other hardware or software related errors compromising packet forwarding. Other syslog events need to be examined to come to the trigger of such host loopback wedge conditions.

In the output of "show log messages" review the events that occurred at or just before the appearance of the "HOST_LOOPBACK_MAKE_CMERROR_ID" alarm. Frequently these events help identify the cause.

Recommended actions:

  1. No RMA required.
  2. By default, disable-pfe action will be taken if major error alarm is raised.
  3. An FPC reset during a maintenance window is needed to clear this error and bring the PFE back into service.

If the issue is seen repeatedly even after restarting the FPC, contact Support for further investigation and assistance.

Below is the one of the real world example of how the corrupted packets can cause host loopback wedge and other errors:

Sep 15 17:37:55.425 2024 device : %PFE-3: fpc3 XMCHIP(0): XMCHIP(0): DDRIF: Checksum error for WO0 - Channel 4, Address 0x2, Checksum Errors 68, Checksum Poison Count 0
Sep 15 17:37:55.427 2024 device : %PFE-3: fpc3 XMCHIP(0): XMCHIP(0): DDRIF: 2x32 Instance 1, Part 3 - Write rate: 3997266, Read rate: 3997202, Total rate: 7994468
Sep 15 17:37:55.484 2024 device : %PFE-3: fpc3 XMCHIP(0): XMCHIP(0): DDRIF: Packet memory - Write rate: 70671879, Read rate: 70671064, Total rate: 141342943
Sep 15 17:37:55.631 2024 device : %PFE-3: fpc3 XMCHIP(0): XMCHIP(0): DDRIF: Linkram memory - Write rate: 0, Read rate: 0, Total rate: 0
Sep 15 17:37:55.757 2024 device : %PFE-3: fpc3 XMCHIP(0): XMCHIP(0): MALLOC: DREF count underflow
Sep 15 17:37:55.844 2024 device : %PFE-3: fpc3 XMCHIP(0): XMCHIP(0): WO0: Packet error - Error Packets 2, Stream 9
Sep 15 17:37:58.131 2024 device : %PFE-3: fpc3 Cmerror Op Set: XMCHIP(0): DDRIF: Checksum error detected - Channel 4, Rate 68 per second  (URI: /fpc/3/pfe/0/cm/0/XMCHIP(0)/0/XMCHIP_CMERROR_DDRIF_INT_REG_CHKSUM_ERR_MINOR)
Sep 15 17:38:09.550 2024 device : %PFE-3: fpc3 Cmerror Op Set: XMCHIP(0): XMCHIP(0): WO0: Cell packing interface error - data32_err_log 0x268f2489  (URI: /fpc/3/pfe/0/cm/0/XMCHIP(0)/0/XMCHIP_CMERROR_WO_INT_REG_PACK_ERR)
Sep 15 17:38:13.759 2024 device : %PFE-3: fpc3 Cmerror Op Set: XQCHIP(46): XQ-chip[0]: DROP sched dequeue underflow  (URI: /fpc/3/pfe/0/cm/0/XQCHIP(46)/0/XQ_CMERROR_DROP_INT_REG_SCHED_UNDERFLOW)
Sep 15 17:38:17.150 2024 device : %PFE-3: fpc3 Cmerror Op Set: XQCHIP(46): XQ-chip[0]: Packet-Reader fast eop request was made to an empty queue  (URI: /fpc/3/pfe/0/cm/0/XQCHIP(46)/0/XQ_CMERROR_PR_INT_REG_FEOP_DRY)
Sep 15 17:38:20.563 2024 device : %PFE-3: fpc3 Cmerror Op Set: XQCHIP(46): XQ-chip[0]: Packet-Reader slow eop request was made to an empty queue  (URI: /fpc/3/pfe/0/cm/0/XQCHIP(46)/0/XQ_CMERROR_PR_INT_REG_SEOP_DRY)
Sep 15 17:38:24.121 2024 device : %PFE-3: fpc3 Cmerror Op Set: XQCHIP(46): XQ-chip[0]: CPQ dry queue error - fast eop request received for an empty queue  (URI: /fpc/3/pfe/0/cm/0/XQCHIP(46)/0/XQ_CMERROR_CPQ_INT_REG_DRY_DEQUEUE_ERROR)
Sep 15 17:38:27.328 2024 device : %PFE-3: fpc3 Cmerror Op Set: Host Loopback: HOST LOOPBACK WEDGE DETECTED IN PATH ID 0  (URI: /fpc/3/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[0])
Sep 15 17:38:30.597 2024 device : %PFE-3: fpc3 Cmerror Op Set: Host Loopback: HOST LOOPBACK WEDGE DETECTED IN PATH ID 2  (URI: /fpc/3/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[2])
Sep 15 17:38:33.787 2024 device : %PFE-3: fpc3 Cmerror Op Set: Host Loopback: HOST LOOPBACK WEDGE DETECTED IN PATH ID 4  (URI: /fpc/3/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[4])
Sep 15 17:38:37.063 2024 device : %PFE-3: fpc3 Cmerror Op Set: Host Loopback: HOST LOOPBACK WEDGE DETECTED IN PATH ID 6  (URI: /fpc/3/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[6])
Sep 15 17:38:40.248 2024 device : %PFE-3: fpc3 Cmerror Op Set: Host Loopback: HOST LOOPBACK WEDGE DETECTED IN PATH ID 8  (URI: /fpc/3/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[8])

Modification History

2024-09-23: Major formatting changes
2023-10-11: Updated Categories
2022-07-08: Added internal notes