Description

Multiple ports on a working FPC went down after the ungraceful removal of another FPC. Customer performed L1 troubleshooting like Cleaning Fiber/Optics, Replacing Fiber/Optics, testing by providing hard loops to the ports, etc... but none of them helped in recovery. 

Symptoms

One of the working FPC was removed by customer in a maintenance window without powering it off. This has caused fatal errors like below on another FPC#PFE 

Nov 18 07:17:20 2023  labrouter fpc0 Error: /fpc/0/pfe/0/cm/0/PE_Chip/0/PECHIP_CMERROR_FI_MEM_MISC_PROTECT_FSET_REG_DETECTED_GNTPIV2PPFE_MEM1 (0x210744), scope: pfe, category: io, severity: fatal, module: PE Chip, type: Description for PECHIP_CMERROR_FI_MEM_MISC_PROTECT_FSET_REG_D

Nov 18 07:17:20 2023  labrouter fpc0 Performing action disable-pfe for error /fpc/0/pfe/0/cm/0/PE_Chip/0/PECHIP_CMERROR_FI_MEM_MISC_PROTECT_FSET_REG_DETECTED_GNTPIV2PPFE_MEM1 (0x210744) in module: PE Chip with scope: pfe category: io level: fatal 

Nov 18 07:17:20 2023  labrouter fpc0 ISSUING ifd-down for et-0/0/0 (243) on pfe-0 
Nov 18 07:17:20 2023  labrouter fpc0 ISSUING ifd-down for et-0/0/2 (245) on pfe-0 
Nov 18 07:17:21 2023  labrouter fpc0 ISSUING ifd-down for et-0/0/4 (247) on pfe-0 
Nov 18 07:17:21 2023  labrouter fpc0 ISSUING ifd-down for et-0/0/6 (249) on pfe-0 
Nov 18 07:17:21 2023  labrouter fpc0 ISSUING ifd-down for et-0/0/8 (251) on pfe-0 

Nov 18 07:17:21 2023  labrouter fpc0 pechip_set_pfe_disabled:976: PECHIP[0]: PFE Marked Disabled !!! 
Nov 18 07:17:21 2023  labrouter fpc0 pechip_set_pfe_disabled:983: PECHIP[0]: Deregisterd TOE PIO Vectors !!! 
Nov 18 07:17:22 2023  labrouter fpc0 JPRDS_FDB:ERR:jprds_fdb_set_pfe_disabled(),8545: pfe 0 marked disabled  

Nov 18 07:17:22 2023  labrouter fpc0 Cmerror Op Set: PE Chip::FATAL ERROR!! from PE0[0]: FI:mem_misc_protect intr: 0x00001000: GntPipe V2PPFE table mem1 MEM:vpfe2ppfe_mapping, MEMTYPE:5, ADDR :[he791d4c,he791d4c], LOG_ADDR :[2516,2516], WIDTH :[32],  

Nov 18 07:18:42 2023  labrouter fpc0 Performing action log for error /fpc/0/pfe/0/cm/0/PE_Chip/0/PECHIP_CMERROR_FI_MEM_MISC_PROTECT_FSET_REG_DETECTED_GNTPIV2PPFE_MEM1 (0x210744) in module: PE Chip with scope: pfe category: io level: fatal 
Nov 18 07:18:42 2023  labrouter fpc0 Cmerror Op Clear: PE Chip::FATAL ERROR!! from PE0[0]: FI: Clear GntPipe V2PPFE table mem1 MEM:vpfe2ppfe_mapping, MEMTYPE:5, ADDR :[he791d4c,he791d4c], LOG_ADDR :[2516,2516], WIDTH :[32],  
 

Solution

This is a transient condition where the fatal errors on FPC lead to PFE disable action, hence interface on the affected FPC#PFE are staying in down state. 

FPC#PFE can be recovered by rebooting the affected FPC

Modification History

2023-11-30 : Article Created