Sometimes, customers using the Apstra NOS upgrade tool may see some devices finishing the software upgrade without any issues, but other devices end up with issues.

Some devices finish the NOS upgrade with no issues, but on other devices, the Apstra controller is not able to finish the NOS upgrade.

During the NOS upgrade process, the customer can check the upgrade job log. If the customer sees the SSH commands failing with return code 255, as below:

2024-10-17 11:54:06,867 INFO:Command (timeout-120): request system storage cleanup no-confirm
2024-10-17 11:54:06,878 ERROR:Failure reason: , return-code: 255 <----------
2024-10-17 11:54:06,878 ERROR:FAILED 
2024-10-17 11:54:06,878 INFO:Deleting Existing NOS image file /var/tmp/jinstall-host-qfx-5e-x86-64-23.4R2-S3.3-secure-signed.tgz from device
2024-10-17 11:54:06,878 INFO:Command (timeout-120): file delete /var/tmp/jinstall-host-qfx-5e-x86-64-23.4R2-S3.3-secure-signed.tgz
2024-10-17 11:54:06,891 INFO:
2024-10-17 11:54:06,891 ERROR:Failed command: request system storage cleanup no-confirm | no-more
2024-10-17 11:54:06,891 ERROR:Remote ssh command failed

The log above indicates that the Apstra controller could not get the SSH command response property. To avoid these issues, the customer should check/implement the following changes:

a) The Firewall devices. The customer needs to check if there is a Firewall between the devices and the controller. Even with the SSH allowed, the firewall can also inspect the SSH session and block some SSH messages. In this case is highly recommended to check the firewall logs and allow SSH between the Apstra controller and the device without the Firewall intervention. 

b) The Tacacs server. As documented here: https://www.juniper.net/documentation/us/en/software/apstra6.1/apstra-user-guide/topics/concept/device-aaa.html the credentials used by the Apstra controller to access the Junos offbox system agent user must always be valid and available. When using the AAA framework we recommend that you add a local user to devices and use it for Apstra offbox system agents. Always have “password” be first in Junos config for authentication order to ensure that the user and commands required by Apstra are authenticated and authorized. In this case is important to have the authentication order configured as: "authentication-order [ password radius ]".

c) The SSH limits configurations. As documented here: https://www.juniper.net/documentation/us/en/software/apstra6.1/apstra-install-upgrade/topics/ref/apstra-server-communication-ports.html the Apstra controller requires a minimum of eight (8) SSH connections, two (2) SSH max-sessions-per-connection, and twenty (20) SSH rate-limit (maximum number of connection attempts per minute). if the customer has different SSH configurations,  is highly recommended to increase the SSH limit number as other users may are also accessing the devices and consuming the SSH resources. 

2024-10-23 : Article Created

2026:04:09 : Updated documentation link to latest Apstra-6.1.1 ( Deepak )

2024-1017-288635 Do not Publish 2024-1017-288635 2024-1017-288635 2024-1017-288635

2024-1017-288635 2024-1017-288635

2024-1017-288635