How do I troubleshoot device that was UP in Junos space but is now showing down.

The following KB lists some areas to check if a Device is not connected to Junos Space.
Open a Support case for further assistance if needed.

Identify Device Connection method

Troubleshooting process differs depending on how the Device-Space connection is initiated

• Space initiated
• Device initiated
  • Device has "system services outbound-ssh" config with Junos Space IP address

From Junos Space WebUI, Space Platform Device list, Look for "Device Connection Method" column.  You may need to add the column (click on the arrow on any existing column)

Common Issues

1. User Credential problems (Did the password Space uses to connect to device change on device (or remote auth server device uses)
   • KB78447 [juniper.net]
2. Key-based authentication devices are down with "Key Conflict" or "Key Conflict-Unverified"
   • Key-based authentication devices are down with "Key Conflict" or "Key Conflict-Unverified" error after upgrade to Junos Space 21.1

Space Initiated Connection Specific Issues

• Confirm Junos Space can successfully SSH from Space Server CLI  to device
  • Check connectivity from all space nodes.
  • After logging into Space CLI, attempt to SSH to the device.

Device Initiated Connection Specific issues

Junos Device initiates a outbound-ssh connection to Junos Space

• Check if connection is established, continually trying to connect, or not connecting at all
  • for Established 7804 connection on device
    • show system connections
  • Check for established 7804 connection for device on Junos space CLI (Remember to check all Junos Space Jboss nodes)
    • netstat -an | grep 7804 | grep 10.10.10.10
    • Note: Multiple devices may be connecting from same remote IP address (NAT)
  • Confirm port # match
  • Wait 2-5 mins and check again.
  • If port number is changing, this shows we fail to connect completely and retry
  • if port number is not changing, Junos Space has marked the device as down, but the Device did not disconnect completely
    • This could be a defect on Junos Space or the device
    • Workaround (Cause Device to disconnect/reconnect to space)
      • Run Commands on Junos device:

        • deactivate system services outbound-ssh

        • commit

      • Wait about 1 minute
        • Alternatively, confirm the netstat or show system connections no longer show device session
      • Run Commands on Junos device

        • activate system services outbound-ssh

        • commit

      • Device should reconnect
  • If port number IS changing, the device is trying to reconnect and is rejected by Junos space for some reason
    • Look for device IP and deviceID (from device outbound-ssh config) in server.log.  make sure to check all Junos Space application Logic nodes
      • tail -F /var/log/jboss/servers/server1/server.log | grep 10.10.10.10
      • tail -F /var/log/jboss/servers/server1/server.log | grep DEVICE_ID
    • Possible reasons that may appear for connection failing
      • DeviceID not allowed to connect
        • Invalid (ID does not exist in Space)
        • Duplicate (another/maximum device already connected with ID)
      • Known issue in Space 23.1, resolved with 23.1R1-hotpatch-v7 PR 1803171

Multiple Junos Space Application Logic nodes

Ensure Junos Space fabric status shows both nodes as UP and are not loading schema on the Administration, Fabric page
Check Device load balancing

• If you have a large number of devices (1000s) connected to each Junos Space node, allow 30m-1h after restart and connections + resync device operations to complete.
• If you have any system performing frequent Junos Space API operations during restart/device discovery.  It may take longer for all devices to show as UP and sync. Temporarily disable API operations when Space is restarting