This article describes the procedure to recover a Juniper Apstra VM from a corrupted file system. When the file system is corrupted, it may no longer be possible to start the Juniper Apstra services. The fsck "check & repair" Linux utility must be run in this case to attempt to recover the file system.
fsck
NOTE: fsck is not guaranteed to repair the file system, and if it does not, there is no alternative but to re-create the entire Juniper Apstra environment. For this reason, users MUST create regular backups and copy them off the Juniper Apstra VM to a safe location. Refer to Juniper Apstra documentation for details about creating backups.
Juniper Apstra services are not running.
On the VM's CLI, boot messages indicate an issue with the EXT4 file system.
These messages are also visible in /var/log and dmesg output.
/var/log
dmesg
File systems can sometimes get corrupted. The cause may be hardware related, or due to incorrect shutdown of the VM. This is generic to Linux, and not specific to Juniper Apstra alone.
In order to recover, the fsck utility must be run. If the disk is mounted in read-write mode (RW), fsck cannot repair any file system issues. In order to use fsck to repair the file system, the VM must be booted into Recovery Mode and the file system must be re-mounted in read-only mode (RO). When this is done, fsck can be run. Use the following steps to achieve this:
Reboot the VM into the boot loader (grub). Depending on your environment, this is done by holding the left Shift key or the F2 key when booting.
From the "GRUB" menu, select " Advanced Options ":
Enter the GRUB credentials: admin / apstra
admin
apstra
Select an entry with "recovery mode":
Select the option to drop to the root shell:
Remount the file system in read-only mode (for example /dev/mapper/aos--server--vg-var+log ).
/dev/mapper/aos--server--vg-var+log
Note: If aos--server--vg-var needs to be checked, the aos--server--vg-var+lib+aos+db and aos--server--vg-var+log mount points must be remounted in RO first:
aos--server--vg-var
aos--server--vg-var+lib+aos+db
aos--server--vg-var+log
$ mount -o remount,ro /dev/mapper/aos--server--vg-var+log
Run fsck to check the file system. If errors are encountered, fsck will try to correct them:
$ fsck -y /dev/mapper/aos--server--vg-var+log
Type "exit" and select "resume" from the Recovery menu to continue booting the VM as normal.