Description

This article details the steps to take to debug an SDWAN spoke device that shows a status of Operational DOWN in Juniper CSO SDWAN.

Symptoms

  • Device Provisioned status shows UP in CSO.

  • Device Operational Status shows DOWN in CSO.

  • Device is processing traffic for customer.

  • Customer has CLI access apart from CSO UI access, which shows that the device is accessible.

  • Customer is able to run all commands on the device.

  • CPU status is normal.

Solution

Note: The device in question is an NFX250, which runs in three domains --- WAN(0/1) (JCP <> JDM <> GWR).

JDM hosts the telemetry agent that collects data from GWR and builds a 443-SSH connection towards CSO via the OAM IPsec tunnel, which enables CSO to display operational stats and status.

Steps

  1. Check the JDM shell to see whether the "telemetry agent is running".

    1. It was running in this case.

  2. If 1 is true, check whether the telemetry agent is able to collect stats and is able to connect to CSO to publish the collected data in cloud.

    1. Data was being written to the agent.log file in this case.

  3. Log in to GWR and check whether GWR is able to connect to CSO on ports 7804 and 443.

    1. In this case, we saw that the GWR was trying to but was not able to connect to CSO. (The command that helped to see whether GWR was trying to connect to the CSO: show security flow session destination-port 7804 | refresh 2).

    2. We checked routing and saw an ECMP path with two ST tunnels as next-hop.

    3. We inspected another session for the same tunnel interface by using the command show security flow session interface st0.xxxx.

    4. Given this, we concluded that one of the tunnels had two-way traffic for a majority of the sessions and the other tunnel only had one-way traffic.

    5. We then checked the CSO jobs for any failed jobs for the said spoke site and saw a WAN1 job failure, which left WAN1 in a partially deployed state.

    6. To allow connection toward the CSO and get it back online, we deactivated the stuck tunnel, which uses the WAN that was in a partially deployed state (command: set interface st0.xxxx disable).

Modification History

2023-12-11: Article Created