Description

This article explains why a QFX device might reboot in Amnesiac mode after upgrade to Junos OS Release 19.3R3.6 from an earlier version. Such a reboot is seen to happen when Junos OS code is upgraded from an initial version where a certain configuration statement is not required/mandatory to a target Junos OS version where it is.

Symptoms

Suppose that a QFX switch is running Junos OS Release 17.3R3.10 or 18.4R1.8 with the MC-LAG configuration where the statement redundancy-group within the mc-ae interface stanza is not mandatory and thus not enforced by commit check.

Now consider that you upgrade the same QFX switch to a version, for example Junos OS Release 19.3R3.6, where the statement redundancy-group within the mc-ae interface stanza is mandatory and enforced.

After the switch is upgraded and reboots, the switch will boot up in Amnesiac mode.

This was tested in the lab with the same results on the following upgrades:

  • Junos OS Release 17.3R3.10 directly to Release 19.3R3.6

  • Junos OS Release 18.4R1.8 directly to Release 19.3R3.6

Solution

The mgd (management) daemon is unable to commit the configuration after the switch reboots, because the configuration is missing mandatory statements.

Due to this, the switch loads a partial configuration and boots in Amnesiac mode.

Refer to the release notes for Junos OS Release 19.3R3 , where a new commit check has been introduced for redundancy-group (on page 48).

After the switch reboots, the mgd daemon will commit the configuration as the last step. You will see that it fails and therefore boots up in Amnesiac mode, without any configuration:

(Output of code installation and reboot snipped for brevity; only last few lines from console output shown) 

pci-hgcomdev module loadedhw.dcf.flex_nonflex: 0 -> 1
kern.timecounter.hardware: TSC -> TSC
*** Creating PVIDb..\n
846+0 records in
846+0 records out
439920 bytes transferred in 0.009735 secs (45191045 bytes/sec)
Copied libschema-filter-dd.tlv to /usr/lib/dd/filter\n
/dev/md38: 1024.0MB (2097152 sectors) block size 32768, fragment size 4096
        using 4 cylinder groups of 256.03MB, 8193 blks, 32896 inodes.
        with soft updates
super-block backups (for fsck_ffs -b #) at:
 192, 524544, 1048896, 1573248
mount_cd9660: /dev/gpt/junos: Invalid argument
Junosprocfs mounted on /junosproc.
@ 1624895407 [2021-06-28 15:50:07 UTC] mgd start
Creating initial configuration:  ...
<output>

[edit interfaces ae1 aggregated-ether-options]
  'mc-ae'
    Missing mandatory statement: 'redundancy-group'
[edit interfaces ae2 aggregated-ether-options]
  'mc-ae'
    Missing mandatory statement: 'redundancy-group'
[edit interfaces ae4 aggregated-ether-options]
  'mc-ae'
    Missing mandatory statement: 'redundancy-group'
[edit interfaces ae6 aggregated-ether-options]
  'mc-ae'
    Missing mandatory statement: 'redundancy-group'

mgd: error: commit failed: (missing mandatory statements)
Warning: Failed to commit active configuration.
Warning: Trying to commit recovery mode configuration.
<output>
[edit interfaces ae1 aggregated-ether-options]
  'mc-ae'
    Missing mandatory statement: 'redundancy-group'
[edit interfaces ae2 aggregated-ether-options]
  'mc-ae'
    Missing mandatory statement: 'redundancy-group'
[edit interfaces ae4 aggregated-ether-options]
  'mc-ae'
    Missing mandatory statement: 'redundancy-group'
[edit interfaces ae6 aggregated-ether-options]
  'mc-ae'
    Missing mandatory statement: 'redundancy-group'

mgd: error: commit failed: (missing mandatory statements)
Warning: Commit failed, activating partial configuration.
Warning: Edit the router configuration to fix these errors.
@ 1624895424 [2021-06-28 15:50:24 UTC] mgd done
@ 1624895442 [2021-06-28 15:50:42 UTC] mountlater done
Starting cron.

Mon Jun 28 15:50:43 UTC 2021

FreeBSD/amd64 (Amnesiac) (ttyu0)

login: root
Jun 28 16:03:21   login: login_getclass: unknown class 'junos-login-defaults'
Last login: Mon Jun 28 15:05:15 on ttyu0

--- JUNOS 19.3R3.6 Kernel 64-bit FLEX JNPR-11.0-20200710.a76c78d_buil
root@:RE:0%

Important: Make sure that you read and understand the target code release notes before upgrade. Further, attempt an upgrade in the lab before you carry it out in production to avoid unexpected results from minor code variations between releases.

After the switch boots in Amnesiac mode, no SSH access should be possible. Therefore, log in via console as root .

You will be able to commit the candidate configuration as-is, and regain remote access to the switch.

However, to prevent this issue from occurring again when the switch reboots the next time, you should add the missing mandatory configuration statements and commit: 

{master:0}[edit]
root# show | compare
[edit]
- version 18.4R1.8;
+ version 19.3R3.6; 

You will notice that the original configuration is still present as the candidate configuration: 

{master:0}[edit]
root# show
## Last changed: 2021-06-28 15:58:31 UTC
version 19.3R3.6;
apply-groups default;
system {
    host-name SWITCH;
    time-zone Europe/Brussels;
}
chassis {
    aggregated-devices {
        ethernet {
            device-count 60;
        }
    }
}
interfaces {
    et-0/0/0 {
        ether-options {
            802.3ad ae0;
        }
    }                                 
(snip)
 
root# wildcard range set interfaces ae[0-2,4,6] aggregated-ether-options mc-ae redundancy-group 1

{master:0}[edit]
root# commit and-quit
configuration check succeeds
commit complete
Exiting configuration mode