In some cases, a Routing Engine running Junos with FreeBSD 10 kernel (Junos OS 15.1+) may encounter a file system corruption. This can happen in either the primary partition residing in the hard drive (labeled /junos ), or in the recovery partition residing in the compact flash (labeled /oam ).
/junos
/oam
This article explains the symptoms and suggests a repair procedure using fsck.
The corruption in the primary partition can cause the system to reboot and load Junos OS from the OAM partition, or get stuck in a boot loop if the system is unable to load Junos OS from the OAM partition. This is what you can expect if you have a corruption in the main partition:
Main Menu 1. Boot [J]unos volume 2. Boot Junos volume in [S]afe mode 3. [R]eboot 4. [B]oot menu 5. [M]ore options Choice: Booting from Junos volume ... / Rebooting... / Booting from recovery snapshot ... [...] --- JUNOS 15.1R5-S4.2 Kernel 64-bit JNPR-10.3-20170523.350481_build Note: Junos is currently running in recovery mode on the OAM volume root@router1-re0:~ # root@router1-re0:~ # cli root@router1-re0> show system storage Filesystem Size Used Avail Capacity Mounted on /dev/md0.uzip 21M 21M 0B 100% / devfs 1.0K 1.0K 0B 100% /dev tmpfs 14G 360M 14G 2% /.mount /dev/gpt/oam 3.8G 853M 2.6G 24% /mnt <..>
Corruption in the recovery partition will make it impossible to restore the system in case the primary partition fails, and can also cause a future Junos OS upgrade to fail because it will need the /oam partition. You can check if you have a corrupted /oam partition by using the following CLI command:
user1@router1-re0> show system snapshot No non-recovery snapshots available on the Junos volume Recovery Snapshots:↵ mount: /dev/gpt/oam: Invalid argument
If you attempt an upgrade with a corrupted /oam partition, you will see the following error:
Verified oam signed by PackageProductionEc_2017 mount: /dev/gpt/oam: R/W mount of /.mount/oam denied. Filesystem is not clean - run fsck.: Operation not permitted mount: /dev/gpt/oam: R/W mount of /.mount/oam denied. Filesystem is not clean - run fsck.: Operation not permitted Verified os-compat32 signed by PackageProductionEc_2017 Verified os-crypto signed by PackageProductionEc_2017 Verified os-kernel-prd signed by PackageProductionEc_2017 Verified os-libs-compat32 signed by PackageProductionEc_2017 Verified os-libs signed by PackageProductionEc_2017 Verified os-runtime signed by PackageProductionEc_2017 Verified os-vmguest signed by PackageProductionEc_2017 Verified os-zoneinfo signed by PackageProductionEc_2017 Verified py-base signed by PackageProductionEc_2017 Verified py-extensions signed by PackageProductionEc_2017 Verified vrr-mx signed by PackageProductionEc_2017 ERROR: Failed to add junos-install-mx-x86-64-16.1R4-S4.3.tgz
This procedure will let you recover the affected partition by using the fsck tool.
If the corruption is in the main partition, perform the following:
Reboot the RE that is reporting the error.
Wait until you see the Boot menu and press the Space bar to stop the Autoboot:
Will try to boot from USB Flash Drive Compact Flash Solid State Drive 1 Solid State Drive 2 Network Trying to boot from Compact Flash Main Menu 1. Boot [J]unos volume 2. Boot Junos volume in [S]afe mode 3. [R]eboot 4. [B]oot menu 5. [M]ore options Choice: Autoboot in 3 seconds...
Press the Space bar and enter 5 to go to the Options menu:
Options Menu 1. Recover [J]unos volume 2. Recovery mode - [C]LI 3. Check [F]ile system 4. Enable [V]erbose boot 5. [B]oot prompt 6. [M]ain menu
Enter 3 to choose "Check [F]ile system". You will see the # prompt. Issue the following command:
# fsck_ufs /dev/ada0p2
You will see an output similar to this:
# fsck_ufs /dev/ada0p2 ** /dev/ada0p2 ** Last Mounted on /.mount ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups SUMMARY BLK COUNT(S) WRONG IN SUPERBLK SALVAGE? yes 19141 files, 2921404 used, 2110321 free (1529 frags, 263599 blocks, 0.0% fragmentation) ***** FILE SYSTEM WAS MODIFIED ****
Reboot the system with the reboot command.
reboot
If the corruption is in the OAM partition, perform the following:
user1@router1-re0> start shell user root Password: root@router1-re0:/var/home/user1 # root@router1-re0:/var/home/user1 # fsck_ufs /dev/ada1p2
root@router1-re0:/var/home/user1 # fsck_ufs /dev/ada1p2 ** /dev/ada1p2 ** Last Mounted on /.mount/oam ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups SUMMARY BLK COUNT(S) WRONG IN SUPERBLK SALVAGE? yes 68 files, 455156 used, 533075 free (43 frags, 66629 blocks, 0.0% fragmentation) ***** FILE SYSTEM WAS MODIFIED *****
In some cases, we may have a file system corruption that affects the superblock of the partition:
root@router1-re0:/var/home/user1 # fsck_ufs /dev/gpt/oam ** /dev/gpt/oam Cannot find file system superblock /dev/gpt/oam: CANNOT FIGURE OUT FILE SYSTEM PARTITION
In order to recover from this issue, we can use one of two options:
Option 1:
user1@router1-re0> request system recover oam-volume NOTICE: Recovering the OAM volume ... ada1p2 deleted ada1 created ada1p1 added bootcode written to ada1 ada1p2 added /dev/gpt/oam: 3998.0MB (8187840 sectors) block size 32768, fragment size 4096 using 7 cylinder groups of 626.09MB, 20035 blks, 80256 inodes. super-block backups (for fsck_ffs -b #) at: 192, 1282432, 2564672, 3846912, 5129152, 6411392, 7693632 Verified oam signed by PackageProductionEc_2017 method ECDSA256+SHA256 Installing OAM volume contents ... The OAM volume is now installed NOTICE: Creating a recovery snapshot on the OAM volume ... Creating image ... Compressing image ... Image size is 795MB Recovery snapshot created successfully NOTICE: OAM volume recovered
Option 2:
root@router1-re0:/var/home/user1 # newfs -U /dev/ada1p2 /dev/ada1p2: 3998.0MB (8187840 sectors) block size 32768, fragment size 4096 using 7 cylinder groups of 626.09MB, 20035 blks, 80256 inodes. with soft updates super-block backups (for fsck_ffs -b #) at: 192, 1282432, 2564672, 3846912, 5129152, 6411392, 7693632 root@router1-re0:/var/home/user1 # fsck_ufs -b 192 /dev/ada1p2 Alternate super block location: 192 ** /dev/ada1p2 ** Last Mounted on ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups SUMMARY BLK COUNT(S) WRONG IN SUPERBLK SALVAGE? [yn] y 2 files, 2 used, 988229 free (21 frags, 123526 blocks, 0.0% fragmentation) UPDATE STANDARD SUPERBLOCK? [yn] y ***** FILE SYSTEM IS CLEAN ***** ***** FILE SYSTEM WAS MODIFIED *****
***
2020-06-18: Article reviewed for accuracy, no changes required; article accurate and valid.
2021-05-04: Minor modification in Solution ( "Corruption in Man Partition") part; the remaining content is valid and accurate.