Description

This article will explain about the steps to recover the situation, when both the routing-engines were not able to boot from the primary partition

 

Symptoms

We will see the below active alarm related to both REs:

 

{master}

juniper@MX> show chassis alarms

2 alarms currently active

Alarm time Class Description

2024-04-05 01:22:49 -06 Minor Host 1 Boot from alternate media

2024-04-05 01:22:39 -06 Minor Host 0 Boot from alternate media

Solution

A few of the causes are listed as follows:

 

  • A transient issue with the RE that usually appears after booting
  • An issue with storage media 
  • RE failure

To troubleshoot this problem, perform the following steps:

 

Step 1 - Login to backup routing engine

 

request routing-engine login other-routing-engine

 

Step 2 - try to boot the device from the Junos volume (from backup RE1). If alarms get clear for Host1 with this step, then move to step 5

 

request system reboot junos

  

Step 3 - If the alarm does not clear after the reboot, this means that the device is still booting from the OAM volume . Try to recover junos-volume

 

request system recover junos-volume

 

Step 4 - Reboot RE1, Once RE1 is up again, check if chassis alarm is still present or it has cleared. If alarm still appears then perform Step10, Step11 and Step12

 

request system reboot

 

Step 5 - Make RE1 as master by doing switchover:

 

request chassis routing-engine master switch check   >>> to verify the status of backup re

request chassis routing-engine master switch

 

Step 6 - Login to RE0 and reboot it:

 

request routing-engine login other-routing-engine

request system reboot junos

 

Step 7: Once RE0 is up, check if alarm is present on it has cleared for both REs. If alarm still appears for RE0 then proceed with step 8

  

Step8: Try to recover junos-volume and then proceed with reboot RE0. If alarm still appears then perform Step10, Step11 and Step12

 

request system recover junos-volume

request system reboot

 

Step9: Make RE0 as master by doing switchover:

 

request chassis routing-engine master switch check   >>> to verify the status of backup re

request chassis routing-engine master switch

 

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

 

Step 10:

Reboot RE again and wait until you see the Boot menu. When you see the Boot menu, press the Space bar to stop Autoboot. (Ensure that you have the console connected to this RE just in case you lose access). You will be directed to the boot options menu as shown below:

 

Main Menu               

1. Boot [J]unos volume

2. Boot Junos volume in [S]afe mode                                 

3. [R]eboot   

4. [B]oot menu

5. [M]ore options                 

Choice: Autoboot in 3 seconds...

Enter 5 to go to the Options menu:                                

Options Menu

1. Recover [J]unos volume

2. Recovery mode - [C]LI                 

3. Check [F]ile system               

4. Enable [V]erbose boot                 

5. [B]oot prompt                  

6. [M]ain menu

Enter 3 to choose "Check [F]ile system". Issue the following command:               

# fsck_ufs /dev/ada0p2   

You will see the below output:    

 # fsck_ufs /dev/ada0p2

           ** /dev/ada0p2

           ** Last Mounted on /.mount

 ** Phase 1 - Check Blocks and Sizes

 ** Phase 2 - Check Pathnames

 ** Phase 3 - Check Connectivity

 ** Phase 4 - Check Reference Counts

 ** Phase 5 - Check Cyl groups

 SUMMARY BLK COUNT(S) WRONG IN SUPERBLK

 SALVAGE? yes

19141 files, 2921404 used, 2110321 free (1529 frags, 263599 blocks, 0.0% fragmentation)

                           

***** FILE SYSTEM WAS MODIFIED ****

Run request system reboot on the backup RE.

 

Step 11

If you are still unable to recover the file system, perform  USB recovery .

 

  • Log in as the root user and execute the following:

 

root@host% dd if=/var/tmp/filename of=/dev/da1 bs=1m   <-- Run for 20 seconds and press CNTRL+C to stop

The device writes the installation media image to the USB device:

root@host% dd if=<installation media image>of=/dev/da0 bs=1m

1399+0 records in

1399+0 records out

1466957824 bytes transferred in 394.081902 secs (3722469 bytes/sec)

 

Step 12

If the above steps do not work, it is most likely that there is a hardware issue with the RE and that it might need replacement. Contact Support .

Modification History

2024-04-08 :  Initial Version