This article describes the "PE CHIP :: FATAL ERROR “HMCIF: Link0: HMC Fatal Error” syslog message that is reported on PTX devices, and explains how to recover from the error.
"PE CHIP :: FATAL ERROR “HMCIF: Link0: HMC Fatal Error”
The error is reflected as shown below:
Syslog messages
<Date> <router name> <FPC#> Cmerror Op Set: PE Chip::FATAL ERROR!! from PE1[1]: HMCIF: Link0: HMC Fatal Error <Date> <router name> <FPC#> Cmerror Op Set: PE Chip::FATAL ERROR!! from <PFE ID>: HMCIF: Link#: HMC Fatal Error <Date> <router name> <FPC#> JPRDS_FDB:ERR:jprds_fdb_set_pfe_disabled(),7317: <pfe id> marked disabled <Date> <router name> <FPC#> Cmerror Op Set: Generic HMC::FATAL ERROR!! from HMC#-#-#: eri timeout error
Note that PFE is disabled due to a fatal error as shown in the below output:
<router name> > show chassis fpc errors FPC Level Occurred Cleared Threshold Action-Taken Action # Minor 0 0 10 0 LOG| Major 0 0 1 0 GET STATE|ALARM| Fatal 8 0 1 16 DISABLE PFE Pfe-State: pfe-# -DISABLED
When you log in to the vty of the affected FPC, you see the following:
router-name > start shell pfe network fpc# <FPC#> (PTX10002-re0 vty)# show pechip # PE[1] : ASIC Name: PE1, ASIC ID: 1 Version 2.0 JTAG ID: 692393343 Initialized: Yes Fault/Disabled: Yes >>>> PFE is disabled (Marked down) <FPC#> <router name>(PTX10002-re0 vty)# show cmerror module brief --------------------------------------- Module Name Active Errors --------------------------------------- 1 PCIe Error 0 2 CPU Error 0 3 Eth Port Error 0 4 Host Loopback 0 5 Generic HMC 5 >>>>>>> Active Errors 6 TOE-PE-0:0:0 0 7 TOE-PE-0:0:1 0 8 TOE-PE-0:0:2 0 9 PE Chip 3 >>>>>>> Active Errors 10 TOE-PE-1:0:0 0 11 TOE-PE-1:0:1 0 12 TOE-PE-1:0:2 0 13 TOE-PE-2:0:0 0 14 TOE-PE-2:0:1 0 15 TOE-PE-2:0:2 0 16 TOE-PE-3:0:0 0 17 TOE-PE-3:0:1 0 18 TOE-PE-3:0:2 0 19 TOE-PE-4:0:0 0 20 TOE-PE-4:0:1 0 21 TOE-PE-4:0:2 0 22 TOE-PE-5:0:0 0 23 TOE-PE-5:0:1 0 24 TOE-PE-5:0:2 0 25 BCM Switch 0 26 FPC 0
These are fatal errors in the HMCIF Link between HMC-PFE. Because these errors can affect traffic, they are marked as fatal and the PFE that has these errors is marked as disabled (all ports relating to this PFE will be down).
To clear the alarm, restart the FPC via the Command Line Interface (CLI) by using the following syntax:
request chassis fpc (offline | online | restart) slot slot-number"
After restarting the FPC, make sure that the PFE is enabled:
<FPC#>(PTX10002-re0 vty)# show pechip <PFE ID>
PE[#] : ASIC Name: PE#, ASIC ID: # Version 2.0 JTAG ID: XXXXXX Initialized: Yes Fault/Disabled: No ////////// PFE is not Disabled ( enabled ) <router name>(PTX10002-re0 vty)# show cmerror module brief --------------------------------------- Module Name Active Errors --------------------------------------- 1 PCIe Error 0 2 CPU Error 0 3 Eth Port Error 0 4 Host Loopback 0 5 Generic HMC 0 >>>>> No Active Errors 6 TOE-PE-0:0:0 0 7 TOE-PE-0:0:1 0 8 TOE-PE-0:0:2 0 9 PE Chip 0 >>>>>> No Active Errors 10 TOE-PE-1:0:0 0 11 TOE-PE-1:0:1 0 12 TOE-PE-1:0:2 0 13 TOE-PE-2:0:0 0 14 TOE-PE-2:0:1 0 15 TOE-PE-2:0:2 0 16 TOE-PE-3:0:0 0 17 TOE-PE-3:0:1 0 18 TOE-PE-3:0:2 0 19 TOE-PE-4:0:0 0 20 TOE-PE-4:0:1 0 21 TOE-PE-4:0:2 0 22 TOE-PE-5:0:0 0 23 TOE-PE-5:0:1 0 24 TOE-PE-5:0:2 0 25 BCM Switch 0 26 FPC 0
If the issue is not resolved, open a Technical Service Request with the following logs:
Collect the following information from CLI:
Logs and RSI: For information, refer to Compressing Troubleshooting Logs from /var/logs to Send to Juniper Networks Technical Support .
Output from the following command: <Date> <router name> <FPC#> show chassis fpc errors
<Date> <router name> <FPC#> show chassis fpc errors
Collect the following information from VTY:
<FPC#>(PTX10002-re0 vty)# show pechip <PFE ID> <FPC#>(PTX10002-re0 vty)# show cmerror module brief
2019-12-28: Article reviewed for accuracy. Minor changes made. Article is correct and complete.2020-06-12: Added script to bottom of the Solution.