This article explains why the Packet Forwarding Engine (PFE) may be disabled even after the action "disable-pfe" is removed for all major cmerrors in the chassis. As detailed in this article, the only workaround for this would be to reboot the affected MPC.

Even though the customer has disabled the "disable-pfe" action for all major cmerrors by using the following commands, the PFE still gets disabled when the cmerror occurs. It appears as if there is no disable-pfe action for cmerrors. 

set chassis error major action get-state
set chassis error major action alarm

Jul 10 15:52:46.876 2021  hostname-re0 : %PFE-5: member1-fpc8 Performing action get-state for error /fpc/8/pfe/0/cm/0/LUCHIP(0)/0/CMERROR_LUCHIP_ERROR (0x40008) in module: LUCHIP(0) with scope: board category: functional level: major
Jul 10 15:53:39.245 2021  hostname-re0 : %PFE-5: member1-fpc8 Performing action cmalarm for error /fpc/8/pfe/0/cm/0/LUCHIP(0)/0/CMERROR_LUCHIP_ERROR (0x40008) in module: LUCHIP(0) with scope: board category: functional level: major
Jul 10 15:53:40.717 2021  hostname-re0 : %PFE-5: member1-fpc8 Performing action get-state for error /fpc/8/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[0] (0x20001) in module: Host Loopback with scope: pfe category: functional level: major
Jul 10 15:53:41.226 2021  hostname-re0 : %PFE-5: member1-fpc8 Performing action cmalarm for error /fpc/8/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[0] (0x20001) in module: Host Loopback with scope: pfe category: functional level: major
Jul 10 15:53:42.695 2021  hostname-re0 : %PFE-5: member1-fpc8 Performing action get-state for error /fpc/8/pfe/0/cm/0/TOE-MQ-0:0:0/0/TOE_ERR_TX_BLOCKED[1] (0x10409) in module: TOE-MQ-0:0:0 with scope: pfe category: processing level: major
Jul 10 15:53:43.103 2021  hostname-re0 : %PFE-5: member1-fpc8 Performing action cmalarm for error /fpc/8/pfe/0/cm/0/TOE-MQ-0:0:0/0/TOE_ERR_TX_BLOCKED[1] (0x10409) in module: TOE-MQ-0:0:0 with scope: pfe category: processing level: major
Jul 10 15:54:29.260 2021  hostname-re0 : %PFE-5: member1-fpc8 Performing action get-state for error /fpc/8/pfe/0/cm/0/CM[0]/0/CM_CMERROR_FABRIC_SELFPING (0x2e0006) in module: CM[0] with scope: board category: functional level: major
Jul 10 15:54:29.647 2021  hostname-re0 : %PFE-5: member1-fpc8 Performing action cmalarm for error /fpc/8/pfe/0/cm/0/CM[0]/0/CM_CMERROR_FABRIC_SELFPING (0x2e0006) in module: CM[0] with scope: board category: functional level: major


Jul 10 15:54:30.502 2021  hostname-re0 : %PFE-5: member1-fpc8 PFE 0: 'PFE Disable' action performed. Bringing down ifd xe-20/0/0 841
Jul 10 15:54:30.712 2021  hostname-re0 : %PFE-5: member1-fpc8 PFE 0: 'PFE Disable' action performed. Bringing down ifd xe-20/0/1 842
Jul 10 15:54:30.822 2021  hostname-re0 : %PFE-5: member1-fpc8 PFE 0: 'PFE Disable' action performed. Bringing down ifd xe-20/1/0 843
Jul 10 15:54:31.077 2021  hostname-re0 : %PFE-5: member1-fpc8 PFE 0: 'PFE Disable' action performed. Bringing down ifd xe-20/1/1 844

The PFE is disabled due to a fabric self ping blackhole rather than the cmerror and there is no way to disable this feature. A self ping is used to detect connectivity of a PFE to itself. For example, for MPC10, it is sent from LUSS back to itself, via the Fabric plane MQSS. The rate is 50ms (20pps).

When you use the following commands to simulate a self ping blackhole under a PFE, for example:

test fabric simulate_error enable
test fabric simulate_error blackhole 1 <<<<< 1 is the PFE ID.

Jul 10 15:54:27.972 2021  hostname-re0 : %PFE-3: member1-fpc8 CMTFPC: Reporting self ping blackhole for FPC 8 PFE 0 tx 538274518 rx 538274518 last-rx 538274518
Jul 10 15:54:28.823 2021  hostname-re0 : %PFE-5: member1-fpc8 Error: /fpc/8/pfe/0/cm/0/CM[0]/0/CM_CMERROR_FABRIC_SELFPING (0x2e0006), scope: board, category: functional, severity: major, module: CM[0], type: MPC fabric selfping blackhole
Jul 10 15:54:29.260 2021  hostname-re0 : %PFE-5: member1-fpc8 Performing action get-state for error /fpc/8/pfe/0/cm/0/CM[0]/0/CM_CMERROR_FABRIC_SELFPING (0x2e0006) in module: CM[0] with scope: board category: functional level: major
Jul 10 15:54:29.647 2021  hostname-re0 : %PFE-5: member1-fpc8 Performing action cmalarm for error /fpc/8/pfe/0/cm/0/CM[0]/0/CM_CMERROR_FABRIC_SELFPING (0x2e0006) in module: CM[0] with scope: board category: functional level: major

To restore the disabled PFE, the only way is to reboot the affected MPC. 

2024-02-19: Minor non-technical changes