This article explains what to check for and how to resolve the issue when users encounter the "Stage-one configuration apply on device failed" error in Contrail Service Orchestration (CSO) during bootstrap.
Note: In CSO, a site or CPE refers to an NFX device.
Users may find that the site onboarding in CSO has failed. The job logs show that the bootstrap job is reporting the following error:
<error snip from job logs in CSO UI >
Perform the following steps to resolve the problem:
Check whether the device has established a TCP connection over port 7804 with CSO.
root@jdm:/var/home/juniper# netstat -antp | grep 7804 tcp 0 0 <DEVICE IP>:53214 <CSO IP?:7804 ESTABLISHED</pre>
The above output shows that it is in established state.
/var/log/syslog
As seen below, the Phone Home Client (PHC) has been initiated and is able to reach CSO but it is not able to log in to vJunos0 (JCP component of NFX device). This message is logged continuously.
Feb 4 09:38:37 jdm jdmd: Delete all IFDs Feb 4 09:38:37 jdm jdmd: IFD Constructor: {'ifd_name': 'hsxe0', 'ifd_namespace': 'host'} Feb 4 09:38:37 jdm jdmd: Delete IFD with input: {'ifd_name': 'hsxe0', 'ifd_namespace': 'host'} Feb 4 09:38:37 jdm jdmd: setting 9200 as MTU for interface:hsxe0 Feb 4 09:38:37 jdm jdmd: IFD set for property: ifd_mtu with value 9200 success Feb 4 09:38:37 jdm jdmd: IFD Constructor: {'ifd_name': 'eth0br', 'ifd_namespace': 'host'} Feb 4 09:38:37 jdm phone-home: phcd_apply_config_to_vm: Read:<rpc-reply xmlns:junos="http://xml.juniper.net/junos/18.4R3/junos"> Feb 4 09:38:37 jdm phone-home: phcd_apply_config_to_vm: Read: <rpc-error> Feb 4 09:38:37 jdm phone-home: phcd_apply_config_to_vm: Read: <error-type>application Feb 4 09:38:37 jdm phone-home: phcd_apply_config_to_vm: Read: <error-tag>invalid-value Feb 4 09:38:37 jdm phone-home: phcd_apply_config_to_vm: Read: <error-message>Password expected. Cant login to vjunos0 Feb 4 09:38:37 jdm phone-home: phcd_apply_config_to_vm: Read: <error-severity>error Feb 4 09:38:37 jdm phone-home: phcd_apply_config_to_vm: Read: Feb 4 09:38:37 jdm mgd: UI_CHILD_STATUS: Cleanup child '/usr/sbin/push_nfv_config', PID 31605, status 0 Feb 4 09:38:37 jdm phone-home: phcd_apply_config_to_vm: Read: <cli> Feb 4 09:38:37 jdm phone-home: phcd_apply_config_to_vm: Read: <banner>{master:0} Feb 4 09:38:37 jdm phone-home: phcd_apply_config_to_vm: Read: Feb 4 09:38:37 jdm phone-home: phcd_apply_config_to_vm: Read: Feb 4 09:38:37 jdm mgd: UI_LOGOUT_EVENT: User 'root' logout Feb 4 09:38:37 jdm phone-home: phcd_apply_config_to_vm: Read : Feb 4 09:38:37 jdm phone-home: phcd_apply_config_from_phs: vnf_name:vjunos0 Config commit Failed!
Check the /var/log directory in JCP (vjunos0).
/var/log
As indicated by the following logs, there is an authentication error for jdm-user, which is used by CSO. Due to this error, the bootstrap configuration to the device cannot be committed.
Feb 4 09:16:23 vjunos0 sshd[37534]: Postponed keyboard-interactive for jdm-sysuser from 192.0.2.254 port 52136 ssh2 Feb 4 09:16:23 vjunos0 sshd[37533]: Postponed keyboard-interactive for jdm-sysuser from 192.0.2.254 port 52136 ssh2 [preauth] Feb 4 09:16:23 vjunos0 sshd[37533]: error: PAM: authentication error for jdm-sysuser from 192.0.2.254 Feb 4 09:16:23 vjunos0 sshd: SSHD_LOGIN_FAILED: Login failed for user 'jdm-sysuser' from host '192.0.2.254'
Note: The jdm-sysuser should normally be able to SSH to JCP without any password.
Add the following configuration in JDM operational prompt to enable SSH access without a password for the jdm-sysuser.
request setup jdm-auto-login vjunos0
After adding the above configuration, verify login to JCP (vjunos0) using below command ssh jdm-sysuser@vjunos0 Note: If for any reason "request setup jdm-auto-login vjunos0" doesn't work then zeroize was not done properly for JCP . Its recommended to initiate zeroize again and retry the prestaging and onboarding.
After adding the above configuration, verify login to JCP (vjunos0) using below command
ssh jdm-sysuser@vjunos0
Note: If for any reason "request setup jdm-auto-login vjunos0" doesn't work then zeroize was not done properly for JCP . Its recommended to initiate zeroize again and retry the prestaging and onboarding.
5. Retry the failed job or attempt to activate the site again and verify bootstrap and ZTP jobs are successful.
Added additional step to verify ssh