Description

This article explains the meaning of the "TOE Pkt Xfer:** WEDGE DETECTED IN PFE 0 stream 0 TOE host packet transfer: reason code 0x1" and "Host Loopback:HOST LOOPBACK WEDGE DETECTED IN PFE x" syslog messages that may be seen on MX and T Series routers when the PFE detects traffic congestion, and how to resolve them.

Symptoms

The following log shows that FPC6 has encountered a Major Error:

user@router> show chassis alarms no-forwarding

1 alarms currently active
Alarm time               Class  Description
2018-12-23 08:18:41 ICT  Major  FPC 6 Major Errors 

Alarm set: FPC color=RED, class=CHASSIS, reason=FPC 6 Major Errors
Major alarm set, FPC 6 Major Errors
Major alarm set, FPC 6 Major Errors

fpc6 Error Count reached 0
fpc6 TOE Pkt Xfer:** CLEARING HOST PATH WEDGE ON PFE 0 stream 4 **
fpc6 Host Loopback:Clearing Host Loopback Wedge on PFE path 0
fpc6 Host Loopback:Clearing Host Loopback Wedge on PFE path 1
fpc6 Host Loopback:Clearing Host Loopback Wedge on PFE path 2
fpc6 TOE Pkt Xfer:** WEDGE DETECTED IN PFE 0 stream 4 TOE host packet transfer: reason code 0x1
fpc6 TOE Pkt Xfer:** WEDGE DETECTED IN PFE 0 stream 2 TOE host packet transfer: reason code 0x1
fpc6 TOE Pkt Xfer:** CLEARING HOST PATH WEDGE ON PFE 0 stream 4 **
fpc6 Host Loopback:HOST LOOPBACK WEDGE DETECTED IN PATH ID 2
fpc6 TOE Pkt Xfer:** WEDGE DETECTED IN PFE 0 stream 0 TOE host packet transfer: reason code 0x1
fpc6 TOE Pkt Xfer:** WEDGE DETECTED IN PFE 0 stream 1 TOE host packet transfer: reason code 0x1
fpc6 TOE Pkt Xfer:** CLEARING HOST PATH WEDGE ON PFE 0 stream 2 **
fpc6 Host Loopback:HOST LOOPBACK WEDGE DETECTED IN PATH ID 0
fpc6 Host Loopback:HOST LOOPBACK WEDGE DETECTED IN PATH ID 1

You could also observe below error logs:

Jan 29 19:58:33.483 2014 mte001gcgw00-RE0 fpc0 Host Loopback:HOST LOOPBACK WEDGE DETECTED IN PFE 0
Jan 29 19:58:33.485 2014 mte001gcgw00-RE0 fpc0 Host Loopback:HOST LOOPBACK WEDGE DETECTED IN PFE 1
Jan 29 19:58:33.487 2014 mte001gcgw00-RE0 fpc0 Host Loopback:HOST LOOPBACK WEDGE DETECTED IN PFE 2
Jan 29 19:58:33.563 2014 mte001gcgw00-RE0 fpc0 Host Loopback:HOST LOOPBACK WEDGE DETECTED IN PFE 3

Solution

The Trinity Offload Engine (TOE) wedge condition that blocks the host path for a Packet Forwarding Engine (PFE) may cause service outage to customers. Although a TOE wedge applies only to blocking of host-bound traffic (Control Plane), it is possible that transit traffic may be blocked in scenarios such as bridge-domains/irb. A typical cause for wedge conditions includes memory corruption, traffic patterns, and packet reorder issues.

A Wedge is declared when the PFE detects traffic congestion on the reporting PFE. The traffic that is serviced by the reporting PFE might get affected when this message is logged. In some cases, all traffic to a particular PFE may get black-holed.

When a wedge is indicated, the software attempts to clear the condition caused by the memory read or write errors during the packet processing procedure. This is done to have as few traffic drops as possible.

This event is cleared in the first healing process attempt. If it does not self heal, restart the FPC during a safe window and monitor it for any further errors.
If the errors recur after FPC restart, please contact JTAC for further assistance.

Modification History

  
 2022-10-21 Mark KB as public