Description

For details about the ports used for communication between a 128T Conductor and a Session Smart Router, refer to KB37064 - What ports does a session smart router use to communicate with a Conductor? [juniper.net]

A Session Smart Router can get disconnected from the Conductor or peer routers for various reasons such as underlay network connectivity issues or issues with some internal process.

This article intends to provide some direction towards basic troubleshooting.

Symptoms

Whenever a Session Smart Router gets disconnected from a Conductor or if there is a peer path connectivity issue between two Session Smart Routers, an alarm will be generated on the Conductor UI for the same.

Solution

Before proceeding with the following steps, it might be a good idea to verify all the processes on the Conductor and the Session Smart Router. For checking the status of services running on a 128T Conductor and the Session Smart Router, refer to KB37023 [juniper.net].

Peer Path Connectivity

The Session Smart Routers that peer with one another send BFD packets to monitor their connectivity over the peered links. The Conductor will display peer path connectivity alarms on the UI if any BFD packets are missed over a period of time.

One way to diagnose the problem is by running PCAPs on the SVR path and checking whether the BFD packets are making through to the other end. If the BFD packets do not make through to the other end, then it is an underlay network connectivity issue that must be investigated. If the BFD packets are getting through to the other end, check the /var/log/128technology/routingManager and /var/log/128technology/highway logs on the Session Smart Router nodes for further analysis.

Also run the following command on the router's PLCI to review various BFD related statistics:

#show stats bfd

Asset State

A Session Smart Router connects to the Conductor on TCP ports 930, 4505, and 4506. The Conductor will display alerts if a Session Smart Router gets disconnected over any of the three channels. 

TCP ports 4505/4506 are specific to Salt and are used by Salt Minion on the Session Smart Router to talk to the Salt Server running on the Conductor. This connection is in charge of upgrades, the start/stop/restart process, remote connections, ZTP installations, and plugins on the routers. The 128T process, AutomatedProvisioner, on the Conductor is responsible for all Salt-related activities. 

TCP port 930 is an SSH session that is responsible for configuration, alarms, commands from the Conductor to the router, and so on.

Note: If a router shows as disconnected on AutomatedProvisioner (Salt TCP 4505/4506) but connected on the TCP 930 connection, then it can do most of what it needs to do, just not the few things listed above.

For Salt connectivity issues, check the following:

On the Conductor CLI, run the following commands: 

vi /var/log/128technology/automatedProvisioner logs 
journalctl -f -u salt-minion
sudo t128-salt-key
sudo t128-salt "*" test.ping  (this will do a ping on the salt connection to see if all the routers are reachable)

On the router or Conductor CLI:

journalctl -f -u salt-minion
sudo systemctl status salt-minion