This article defines a procedure to troubleshoot a Stage-One configuration failing to commit on an NFX-250 customer premises equipment (CPE) and reporting the " Stage-one configuration apply on device failed. " error.

In Contrail Service Orchestration (CSO), we see the following Job failure error:

Job Status
Name csp.tssm_bootstrap
User [email protected]
State failed
Actual Start Time 8 Apr 2021, 11:55:45

End Time 8 Apr 2021, 13:44:22
Import Logs [csp.tssm_bootstrap-PT0777-9010-eee1ce02be2841c7a8867c0b8a47e24b]Hierarchical View
Task: bootstrap
8 Apr 2021, 11:55:45Start bootstrap for device [u'default-domain', u'Juniper', u'PT0777-9010']
8 Apr 2021, 11:55:45Bootstrap Task Start, task_id = bootstrap ...
8 Apr 2021, 11:55:46authenticate device in region: regional
8 Apr 2021, 11:55:46Waiting for bootstrap to finish for 6300s
8 Apr 2021, 11:56:12Received activation state: CONFIGURATION_FAILED with activation msg: Stage-one configuration apply on device failed.

When users log in to the NFX Juniper Device Manager (JDM) to check the failure, the following logs are observed in /var/log/syslog .

Apr  8 12:10:10 jdm mgd: UI_LOGIN_CLASS: Failure opening juniper database: Database is already open

Subsequently, when users check the show system commit output, they see multiple commits occurring via netconf.

At the same time, when users try to manually commit, the commit configuration is not accepted even though the commit command completes.

This is because netconf is trying to commit the configuration received from CSO continuously by using the phone-home script.

Since the NFX JDM is the container that is used to manage the device, the bootstrap configuration must be committed on JDM. An NFX device that is managed by CSO will have a script that receives the bootstrap configuration from CSO and commit the same to JDM.

To troubleshoot the problem, investigate the logs in greater detail:

Apr  8 12:10:10 jdm mgd: UI_CFG_AUDIT_SET: User 'root' set: [system host-name] "jdm -> "jdm"
Apr  8 12:10:10 jdm mgd: UI_CFG_AUDIT_SET_SECRET: User 'root' set: [system root-authentication encrypted-password]
Apr  8 12:10:10 jdm phone-home: phcd_apply_config_to_jdm: Read:<nc:rpc-error>
Apr  8 12:10:10 jdm phone-home: phcd_rollback_jdm: wrong configuration leads to JDM configuration rollback
Apr  8 12:10:10 jdm phone-home: phcd_rollback_jdm: written 65 bytes for rollback
Apr  8 12:10:10 jdm phone-home: phcd_rollback_jdm: JDM rollback cmd: /usr/sbin/cli xml-mode netconf < /var/phone-home/phcd_rollback_cfg_rpc
Apr  8 12:10:10 jdm mgd: Warning: main() : no ability to set login as root on linux
Apr  8 12:10:10 jdm mgd: PVIDB: Error retrieving 'global.mchassis_are_vc_mem_fpcs' variable value
Apr  8 12:10:10 jdm mgd: UI_LOGIN_CLASS: Failure to open juniper database: Database is already open
Apr  8 12:10:10 jdm mgd: UI_LOGIN_CLASS: Failure opening juniper database: Database is already open

In the above case, the logs indicate an error as shown above. However, the error is not defined clearly. We can only determine that the configuration after the "system hostname | root-authentication" line is failing.

1. To trace it further, log in to the CSO UI and check the "Stage1 Config" on the Device tab to see if there are any configuration errors. In this case, the user has added an incorrect name-server IP address (8.8..8.8).

2. On the CSO UI, under the correct tenant, navigate to Resources > Site Management > click the site name > click DEVICES.

3. Select the check box on the extreme left to enable the " Stage1 Config " button.

4. Click the Stage1 Config button to view the configuration.

5. To solve this problem, in CSO 5.1.2, delete and re-configure the specified site to activate it again.

Note: If the error observed is different or for a different command, contact Support to investigate the problem further.