Customer inserted replacement of FPC(JNP10K-LC9600) which ran into hard errors like below, FPC replacement keeps rebooting and printed out message below to FPC console:
Model: MX10008
Junos: 21.4R3-S6.1
FPC REV 40 750-114437 JNP10K-LC9600
// FPC alarm //
Nov 27 14:24:37.422 2024 Router craftd[11157]: Major alarm set, FPC 1 Hard errors
// logs printed out to FPC(replacement) console //
root# cty -f fpc1
PXE-E18: Server response timeout.
Press Esc for boot options
Booting ETH00 (02-00-00-00-00-11) ...
Secure boot is enforced
>>Start PXE over IPv4, Press [ESC] to EXIT...
PXE-E21: Remote boot cancelled.
Boot Succeeded -
Booting ETH01 (02-00-00-00-00-11) ...
root #
Please perform the actions below during a maintenance window as a precaution against unexpected issues, such as other FPCs failing to come online after a mastership switchover.
1) For dual-RE platform:
<- Pull out FPC(replacement) from the box
<- Then reboot backup RE(routing engine) so that backup RE is in clean state before mastership switchover.
Notes:
For devices running vmhost-based images, please reboot vmhost rather than request system reboot.
<- After backup RE finishes reboot, then wait for about 10 minutes.
Then performed RE(Routing Engine) switchover if master and backup RE have GRES (Graceful Routing Engine Switchover) and NSR(Non-Stop-Routing) enabled.
Please check output below on both master and backup RE(routing engine) to confirm if master and backup RE are ready to do switchover.
(1) show task replication
Make sure Synchronization Status for all protocols are Complete.
(2) show nonstop-routing
Make sure NSR is enabled.
(3) request chassis routing-engine master switch check
Make sure switchover over is ready
Please check output of cli below on backup RE to make sure switchover is ready.
cli-> show system switchover
<- After performing RE switchover and system settles down, in general, it would take about 10 minutes which depends on route scaling and route roles. you can check cli outputs mentioned in above to check if master and backup RE finishes sync up or not.
Then insert FPC1(replacement) into the box, and check if it can boot up and recover the service or not.
If multiple FPCs encounter boot-up issues, it is better practice to insert one at a time into the router, waiting until the previously inserted FPC comes online without problems before inserting the next.
<- You can login console of FPC to check its output during FPC booting-up.
Enter root shell of JunOS first:
cli-> start shell user root
Then, cty -f fpc# <<< specify fpc number here, for instance, cty -f fpc0
<- If fpc replacement can boot up successfully, Then restarted chassisd on RE0(backup) just in case something was wrong with chassisd which caused FPC bootup issue.
cli-> request routing-engine login backup
If you run above command with root permission, it may drop you in root shell of backup RE, please enter cmd "cli" to go back to cli mode of backup RE.
root-backup_re# cli
cli-> restart chassis-control
2) For single RE(Routing Engine) platform, we have to restart chassisd which will impact service, please do this in maintenance window.
Recommend having console access of the router available during maintenance window, just in case.
(1) Pull out FPC replacement of the router.
(2) Restart chassisd
Note:
Your ssh session to router may be disconnected because all FPCs will restart after restarting chassisd
(3) Then insert the FPC replacement into the router, and check if it could boot up normally, in general this would take up to 5 minutes which depends on FPC type.
(4) If restarting chassisd couldn't resolve the issue, then please reboot whole chassis.
<- For vmhost-based platform:
cli-> request vmhost reboot
<- For legacy JunOS or JunOS-Evo:
cli-> request system reboot