Description

This article explains how to identify the reason of router booting up in Amnesiac mode and steps needed to resolve this issue.

Symptoms

When a device running Junos OS is upgraded with the "no-validate" keyword, sometimes during RE reboot, it may enter an error state and boot with partial config in Amnesiac mode. In this state it's usually not possible to access the device remotely using Telnet or SSH though it may remain reachable and pingable. 

 

When RE is in Amnesiac mode, the below logs or similar are often seen:

Aug 2 08:13:24  authd[18111]: LICENSE_SOCKET_FAILURE: 'evConnect' failed for socket 18:
Aug 2 08:13:29  jpppd: JPPPD_TIMED_OUT_AFTER_WAITING_FOR_SMI: waitSMIInterfacesInitialized: Timed out after waiting 10 seconds for SMI interfaces to initialize 
Aug 2 08:13:34  smid: LICENSE_SOCKET_FAILURE: 'evConnect' failed for socket 8:
Aug 2 08:13:39  jpppd: JPPPD_TIMED_OUT_AFTER_WAITING_FOR_SMI: waitSMIInterfacesInitialized: Timed out after waiting 10 seconds for SMI interfaces to initialize 
Aug 2 08:13:59  last message repeated 2 times
Aug 2 08:14:19  last message repeated 2 times
Aug 2 08:14:24  authd[18111]: LICENSE_SOCKET_FAILURE: 'evConnect' failed for socket 18:

Console logs from the device while it was booting shows that the initial commit failed hence booting with partial config in Amnesia mode:

@ 1690959537 [2023-08-02 06:58:57 UTC] mgd start
Creating initial configuration: ...
<output>
[edit interfaces interface-set TEST1 interface xe-0/1/0:0 unit]
'22'
Interface unit should have configuration defined for adding to interface-set
[edit interfaces interface-set TEST1 interface xe-0/1/0:0 unit]
'23'
Interface unit should have configuration defined for adding to interface-set
[edit interfaces interface-set TEST1 interface xe-0/1/0:0 unit]
'24'
Interface unit should have configuration defined for adding to interface-set
[edit interfaces interface-set TEST1 interface xe-0/1/0:0 unit]
'25'
Interface unit should have configuration defined for adding to interface-set
</output>
mgd: error: commit failed: (statements constraint check failed)
Warning: Failed to commit active configuration.
Warning: Trying to commit recovery mode configuration.
mgd: error: Unable to find the rescue configuration file: /config/rescue.conf
Warning: Commit failed, activating partial configuration.
Warning: Edit the router configuration to fix these errors.

@ 1690959556 [2023-08-02 06:59:16 UTC] mgd done

Lock Manager
RDM Embedded 7 [04-Aug-2006] http://www.birdstep.com
Copyright (c) 1992-2006 Birdstep Technology, Inc. All Rights Reserved.

Unix Domain sockets Lock manager
Lock manager 'lockmgr' started successfully.

Database Initialization Utility
RDM Embedded 7 [04-Aug-2006] http://www.birdstep.com
Copyright (c) 1992-2006 Birdstep Technology, Inc. All Rights Reserved.

Profile database initialized
Set Enhanced BBE Default...
Enhanced BBE Default for mx960 set to... 2
Set Enhanced BBE Default...
Enhanced BBE Default mode for mx960 set to... 0
cp: /var/etc/login.conf: No such file or directory
Set Enhanced BBElag enhanced disabled 0 to 2
Set Enhanced BBE mode to 0
Enhanced arp scale is disabled
Set Inline Services Cookie Mode to Default...
Inline Services Coookie MODE Default for mx960 set to... 2
No core dumps found.
Prefetching /usr/sbin/rpd ...
Prefetching /usr/libexec64/rpd ...
Prefetching /usr/sbin/lacpd ...
Prefetching /usr/sbin/chassisd ...
@ 1690959562 [2023-08-02 06:59:22 UTC] mountlater start
@ 1690959564 [2023-08-02 06:59:24 UTC] mountlater done
rm: etc/smartd_working.conf: Operation not permitted
newsyslog.junos: pid file doesn't exist: /var/run/syslog.pid
Starting jlaunchhelperd.
Invoking jdid_diag_mode_setup.shAug 2 06:59:30 kernel: ---<<BOOT>>---
Aug 2 06:59:30 kernel: Copyright (c) 1992-2020 The FreeBSD Project.
Aug 2 06:59:30 kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Aug 2 06:59:30 kernel: The Regents of the University of California. All rights reserved.
Aug 2 06:59:30 kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Aug 2 06:59:30 kernel: FreeBSD JNPR-12.1-20220523.3155a70_builder_stable_12_212 #0 r356482+3155a706a46(HEAD): Tue May 31 18:44:38 PDT 2022

FreeBSD/amd64 (Amnesiac) (ttyu0)

 

Another example of console logs during boot up when router ended up in Amnesia mode:

@ 1740473288 [2025-02-25 08:48:08 UTC] mgd start
Creating initial configuration: ...
<output> [edit routing-instances R1 protocols bgp group eBGP authentication-algorithm] 'authentication-algorithm md5' May not be configured without authentication-key-chain </output> mgd: error: commit failed: (statements constraint check failed) Warning: Failed to commit active configuration. Warning: Trying to commit recovery mode configuration. mgd: error: Unable to find the rescue configuration file: /config/rescue.conf Warning: Commit failed, activating partial configuration. Warning: Edit the router configuration to fix these errors.
<cut for brevity>
FreeBSD/amd64 (Amnesiac) (ttyu0)

Solution

When the target Junos OS release is based on the newer FreeBSD version than the current Junos OS release, "no-validate" keyword must be used during the upgrade. Changes between Junos OS releases may cause configuration that was working without any issues in the old release to become invalid in the new release.

Recovery options:

  • If there is a single RE in the chassis, login to this RE using root credentials using console connection, and attempt to commit. Commit should fail and reason for the commit failure should appear. 
  • Alternatively, console logs during boot up could be used to identify the reason why the initial commit failed. If no console logs are available from the boot up period, please proceed with RE power cycle to collect them. Errors should be visible between "Creating initial configuration" and "Failed to commit active configuration" lines.

Manually correct the config that is causing the commit failure either by using a good backup config or based on your requirements.

  • If there are two REs in the chassis and one of them is having issues, then "commit synchronize force" or "commit synchronize force full" from the properly working RE can be used. 

Modification History

2025-02-25: Formatting changes

Related Information

https://supportportal.juniper.net/s/article/21-2R2-BGP-incomplete-authentication-algorithm-configuration-statement-will-lead-to-config-check-failed