Description

This article explains about the log message  "CMTFPC: Fabric request time out pfe" & how RE & PFE self heals. 

 

 

 

Symptoms

The below log messages will be seen in device:
 

Sep 2 00:40:31.438 labroot fpc1 CMTFPC: Fabric request time out pfe 0 plane 19 fab_stream 17, attempting recovery

Sep 2 00:40:31.453 labroot fpc8 MQSS(0): FO: Request timeout error - Number of timeouts 6, RC select 1, Stream 17

Sep 2 00:40:31.457 labroot fpc8 MQSS(1): FO: Request timeout error - Number of timeouts 6, RC select 1, Stream 17

Sep 2 00:40:31.463 labroot fpc1 CMTFPC: Fabric request time out pfe 0 plane 20 fab_stream 17, attempting recovery

Sep 2 00:40:31.491 labroot fpc1 CMTFPC: Fabric request time out pfe 0 plane 3 fab_stream 17, attempting recovery

Sep 2 00:40:31.841 labroot fpc8 CMTFPC: Fabric request time out pfe 0 plane 19 fab_stream 17, attempting recovery

Sep 2 00:40:31.841 labroot fpc8 CMTFPC: Fabric request time out pfe 0 plane 20 fab_stream 17, attempting recovery

Solution

The reason for above log messages is due to Packet Forwarding Engines (PFEs) that can send traffic to each other using high-speed links (HSLs). When the fabric has traffic destined to another FPC, it sends out a fabric request to the destination hardware asking for permission or bandwidth to pass the traffic to the destination.

Sometimes there will not be any response from the destination hardware due to some error conditions either on the source hardware or the destination. Then the chassis makes an attempt to self-heal the link between the source and the destination FPC. These messages can occur during an FPC removal/insertion when the destination hardware is congested or under some bad hardware conditions.

 

The syslog messages indicate that there is a link error between the source hardware and destination hardware. The chassis/Routing-Engine initially makes a recovery attempt to self-heal the erroneous links.

 

NOTE: The syslog messages are informational and harmless.

 

The logs have stopped indicates the chassis/Routing-Engine made a recovery attempt to self-heal the erroneous links

 

If the recovery attempt fails, then the alarm might be seen on the device, pointing out the link failure seen on the source and destination hardware . In this case we are not observing any alarms. Monitored the device for 48 hrs.

Please collect below output to check if there is any issue with Plane or link error on FPC.

show chassis fabric fpcs
show chassis fabric plane
show chassis fabric plane-location
show chassis fabric map
show chassis fabric summary
show chassis fabric summary extended
show chassis fabric destinations

Please reach out to JTAC after collecting the details.

Modification History

2024-09-14 : Article Created