Article Detail

Description

Conductor provisioning of managed router nodes (a.k.a. assets) is accomplished using salt. When the conductor is unable to communicate with the salt minion running on a 128T router node, the administrator will be unable to provision or upgrade that node.

Symptoms

The conductor shows the router node (a.k.a asset) in the disconnected state. This condition is rendered a bit differently in the pcli and GUI.

1. From the PCLI:

Enter the pcli as user admin and type the 'show assets' command. Notice that one router node / asset is in the the Disconnected state.

 [email protected]# show assets 
 Wed 2021-06-02 17:57:34 UTC 
 
 ============== =============== ===================== ============== ============== ================ ======== 
  Router         Node            Asset Id              128T Version   Status         Time in Status   Errors 
 ============== =============== ===================== ============== ============== ================ ======== 
  bostonsite1    datacenter1     datacenter-router-1   5.1.0-1.el7    Running        19h 46m 26s           0 
  dallassite1    branchoffice2   branch-2-router       5.1.0-1.el7    Running        19h 46m 27s           0 
  nycsite1       conductor1      conductor             5.1.0-1.el7    Running        19h 54m 41s           0 
  seattlesite1   branchoffice1   branch-1-router       None             Disconnected      31s                   0 
  None           None            datacenter-router-2   None           Pending        19h 56m 24s           0 
 
 Completed in 0.03 seconds 
 [email protected]#

2. From the GUI:

View the DASHBOARD (the first screen after login) depicted below, or the Routers window under Authority . Notice router seattlesite1 and the red circled Disconnected under the Provisioner column:
alt

Solution

1. In some circumstances, it is normal for a router node / asset to be in the Disconnected state for a few minutes or longer:

During an upgrade of 128T software
During an OS reboot of the router node

The router node / asset status displayed by the conductor is updated from status messages sent once every minute by the router node. For this reason, the conductor status may lag behind the actual state of the node.

2. If administrative actions known to to temporarily affect the asset connection have been ruled out, there are several potential causes:

The router node / asset's salt-minion systemd service is not running due to a failure.
The router node / asset has suffered a major failure which affects more than the salt-minion systemd service.
The salt-minion connection is 'inband' (forwarded through the 128T data-plane), and 128T forwarding is not functioning
One or more paths from the router node / asset through the network, to the conductor is impaired or blocked

We will examine each in turn.

3. The router node /asset's salt-minion node is not running due to a failure.
Before triaging further, try restarting the router node's salt-minion systemd service from the conductor PCLI.

First, obtain the router and node name of the affected node / asset:

 [email protected]# show assets 
 Wed 2021-06-02 22:09:53 UTC 
 
 ============== =============== ===================== ============== ============== ================ ======== 
  Router         Node            Asset Id              128T Version   Status         Time in Status   Errors 
 ============== =============== ===================== ============== ============== ================ ======== 
  bostonsite1    datacenter1     datacenter-router-1   5.1.0-1.el7    Running        23h 58m 45s           0 
  dallassite1    branchoffice2   branch-2-router       5.1.0-1.el7    Running        23h 58m 46s           0 
  nycsite1       conductor1      conductor             5.1.0-1.el7    Running        1d 7m 0s              0 
   seattlesite1   branchoffice1     branch-1-router           None           Disconnected   4h 12m 50s            0 
  None           None            datacenter-router-2   None           Pending        1d 8m 43s             0 
 
 Completed in 0.03 seconds

The commands and examel output in the section are based on the 5.1.0 router software.

now issue the command, ' send command reconnect router <router-name> node <node-name> '.

 [email protected]# send command reconnect router  seattlesite1  node  branchoffice1  
 ✔ Restarting minion 1/1 targets complete. 
 Attempted to reconnect: branchoffice1.seattlesite1.

within a minute or perhaps even seconds, the status should change to Connected. The connected status simply means that the router node's salt minion has established a connection to the conductor's salt master.

 [email protected]# show assets 
 Thu 2021-06-03 00:24:24 UTC 
 
 ============== =============== ===================== ============== =========== ================ ======== 
  Router         Node            Asset Id              128T Version   Status      Time in Status   Errors 
 ============== =============== ===================== ============== =========== ================ ======== 
  bostonsite1    datacenter1     datacenter-router-1   5.1.0-1.el7    Running     1d 2h 13m 17s         0 
  dallassite1    branchoffice2   branch-2-router       5.1.0-1.el7    Running     1d 2h 13m 18s         0 
  nycsite1       conductor1      conductor             5.1.0-1.el7    Running     1d 2h 21m 32s         0 
  seattlesite1   branchoffice1   branch-1-router       None            Connected     8s                    0 
  None           None            datacenter-router-2   None           Pending     1d 2h 23m 15s         0 
 
 Completed in 0.03 seconds

While it is possible to use the GUI to follow the asset status by looking at the dashboard (or router) page's router Provisioner [Status] column, 'send command reconnect' and many of the additional tools in this workflow are not available in the GUI

In another minute, if the router node is healthy the status will change to running and the problem is resolved. No further triage is required.

 [email protected]# show ass 
 Thu 2021-06-03 00:25:14 UTC 
 
 ============== =============== ===================== ============== ========= ================ ======== 
  Router         Node            Asset Id              128T Version   Status    Time in Status   Errors 
 ============== =============== ===================== ============== ========= ================ ======== 
  bostonsite1    datacenter1     datacenter-router-1   5.1.0-1.el7    Running   1d 2h 14m 6s          0 
  dallassite1    branchoffice2   branch-2-router       5.1.0-1.el7    Running   1d 2h 14m 8s          0 
  nycsite1       conductor1      conductor             5.1.0-1.el7    Running   1d 2h 22m 21s         0 
  seattlesite1   branchoffice1   branch-1-router       5.1.0-1.el7     Running     4s                    0 
  None           None            datacenter-router-2   None           Pending   1d 2h 24m 4s          0 
 
 Completed in 0.04 seconds

The 'Time in Status' column can be used to keep track of how much time has elapsed since the last change in status. If it has been at least 2 minutes since the reconnect command was issued, and the status is Disconnected then further triage is indicated.

If after 2 minutes, the status is something other than Running or Disconnected, there may be problems with one or more processes on the router node. Triaging this condition is beyond the scope of this document. Please refer to "Understanding and troubleshooting asset status" .

4. Conductor PCLI 'send command reconnect' was unsuccessful (asset status still Disconnected)
If the asset status is still disconnected, it will be necessary to enter the router node or asset's PCLI. It is important to note that many of the commands which will be used to continue triage require further access to the Linux shell as user root.

Establish a terminal session to the router node with disconnected asset status:

Using the connect command from the conductor pcli

From the conductor pcli, use the following command to connect to the remote node:
connect router router-name node node-name username t128

 [email protected]# connect router  seattlesite1 node branchoffice1 
 Connecting... 
 The authenticity of host '[127.127.0.1]:16386 ([127.127.0.1]:16386)' can't be established. 
 RSA key fingerprint is SHA256:FMfsnHxv6Lbssmmipn3Rqk5/gaPmyk+u4tgvJ5O1zGQ. 
 Are you sure you want to continue connecting (yes/no)? yes 
 Warning: Permanently added '[127.127.0.1]:16386' (RSA) to the list of known hosts. 
 [email protected]'s password: 
 FIPS mode initialized. SSH client running in FIPS 140-2 mode 
 Last login: Tue Jun  1 22:20:13 2021 
    ___    ____    _____           ____    _   ____   
   / _ \  |  _ \  |___  |         | __ )  / | |  _ \ 
  | | | | | |_) |    / /   _____  |  _ \  | | | |_) | 
  | |_| | |  _ <    / /   |_____| | |_) | | | |  _ < 
   \___/  |_| \_\  /_/            |____/  |_| |_| \_\ 
                                                      
 [email protected]#

ssh directly to the router node (if permitted)

Use the Conductor GUI to bring up a router node shell session:

5. If a terminal session is established, skip to step 8.

6. If unable to establish a terminal session using one of the above methods, try the others to see if the router node can be connected to at all.

7. If unable to establish any terminal sessions, it is possible the router is experiencing a service-affecting issue.
This is especially if the router node / asset with disconnected asset status is a singleton. Other 128T routers which peer with the disconnected router node may offer some insight as to whether service is affected.
From the conductor PCLI, find other routers which are configured as peers of the disconnected asset / router node) using the following command, where <router-name> is the name of the router with an asset status of Disconnected:

show config run flat | egrep ‘router < router-name> peer.*? name’

 [email protected]# show config run flat | egrep 'router seattlesite1 peer.*? name' 
 ✔ Piping output... 
 config authority router seattlesite1 peer bostonsite1 name            bostonsite1

Next check the peering status of the router with the asset status of Disconnected, from the perspective of each its peers output by the previous command (in this case there is only one peer). <router-name> is the name of the router with asset status of Disconnected and <peer-router-name> is the name of one of the routers it is peered with :

show peers router <peer-router-name</strong> > | egrep <router-name</strong> >

[email protected]# show peers router bostonsite1 | egrep seattlesite1 ✔ Piping output... bostonsite1 -> seattlesite1 datacenter1 bo1 10.0.128.0 up unavailable unavailable bostonsite1 -> seattlesite1 datacenter1 internet 1.1.1.128 up unavailable unavailable [email protected]#

In this case, the router bostonsite1 is actively peering with with seattlesite1 so even though the the router cannot be remotely administered, it showing signs of life.

If, on the other hand the results look like:

[email protected]# show peers router bostonsite1 | egrep seattlesite1 ✔ Piping output... bostonsite1 -> seattlesite1 datacenter1 bo1 10.0.128.0 down unavailable unavailable bostonsite1 -> seattlesite1 datacenter1 internet 1.1.1.128 down unavailable unavailable

Then the router node with the asset status of disconnected cannot be remotely administered and is unable to forward traffic to its peers. Either way a site visit is merited, but the latter scenario is more severe than the former.

STOP HERE

8. ' send command reconnec t' was unsuccessful, but remote terminal session was established.
Let's take a closer look at the salt-minion systemd service running on the router node of interest. To do this it will be necessary to enter enter a Linux shell with root privileges. This does not have to be the root user. Normally the router is installed with a user t128, which has sudo privileges. This is safer than logging in as root.

This is one way to enter the Linux shell as the t128 user. This may not work for you depending on how your system is configured. You will need to have the appropriate credentials for whichever method works on your router:

[email protected]# shell [admin@branch-1-router ~]$ su - t128 Password: Last login: Thu Jun 3 17:43:09 UTC 2021 from gateway on pts/0 [t128@branch-1-router ~]$

Now lets look at the status of the salt-minion systemd service:

[t128t@branch-1-router ~]# sudo systemctl status salt-minion ● salt-minion.service - The Salt Minion Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/salt-minion.service.d └─dnsTimeout.conf, minionWatchdog.conf, restartAlways.conf Active: inactive (dead) since Thu 2021-06-03 18:00:37 UTC; 50m ago Docs: man:salt-minion(1) file:///usr/share/doc/salt/html/contents.html https://docs.saltstack.com/en/latest/contents.html Process: 4528 ExecStart=/usr/bin/salt-minion (code=exited, status=0/SUCCESS) Main PID: 4528 (code=exited, status=0/SUCCESS) Jun 03 17:58:40 branch-1-router salt-minion[4528]: [WARNING ] The function "module.run" is using its deprecated version and will expire in version "Phosphorus". Jun 03 17:58:41 branch-1-router salt-minion[4528]: [WARNING ] The function "module.run" is using its deprecated version and will expire in version "Phosphorus". Jun 03 17:58:55 branch-1-router salt-minion[4528]: [WARNING ] The function "module.run" is using its deprecated version and will expire in version "Phosphorus". Jun 03 17:58:56 branch-1-router salt-minion[4528]: [WARNING ] The function "module.run" is using its deprecated version and will expire in version "Phosphorus". Jun 03 17:59:25 branch-1-router salt-minion[4528]: [WARNING ] The function "module.run" is using its deprecated version and will expire in version "Phosphorus". Jun 03 17:59:27 branch-1-router salt-minion[4528]: 2048 SHA256:jMLuWaTJjpbpWdN/k+V5ysEnz78GDiAW3CRwoRJncUQ branch-1-router (RSA) Jun 03 18:00:36 branch-1-router systemd[1]: Stopping The Salt Minion... Jun 03 18:00:36 branch-1-router salt-minion[4528]: [WARNING ] Minion received a SIGTERM. Exiting. Jun 03 18:00:37 branch-1-router salt-minion[4528]: The Salt Minion is shutdown. Minion received a SIGTERM. Exited. Jun 03 18:00:37 branch-1-router systemd[1]: Stopped The Salt Minion.

Any status other than:

Active: active (running) since Thu 2021-06-03 17:40:15 UTC; 10m ago

indicates that the systemd service is not functioning correctly. If the salt-minion systemd service is active (running) then the problem may not be with the 128 router, it may be the network path between the router and conductor (the salt-minion and salt-master). Please skip ahead to step 9.

It's not clear why the 'send command reconnect' directive would have failed but as the status in the example output shows inactive (dead) lets try to restart the salt-minion systemd service:

[t128@branch-1-router ~]# sudo systemctl restart salt-minion [t128@branch-1-router ~]# sudo systemctl status salt-minion ● salt-minion.service - The Salt Minion Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/salt-minion.service.d └─dnsTimeout.conf, minionWatchdog.conf, restartAlways.conf Active: active (running) since Thu 2021-06-03 18:15:39 UTC; 8s ago Docs: man:salt-minion(1) file:///usr/share/doc/salt/html/contents.html https://docs.saltstack.com/en/latest/contents.html Main PID: 11087 (salt-minion) Tasks: 8 Memory: 68.8M CGroup: /system.slice/salt-minion.service ├─11087 /usr/bin/python3 /usr/bin/salt-minion ├─11094 /usr/bin/python3 /usr/bin/salt-minion └─11096 /usr/bin/python3 /usr/bin/salt-minion Jun 03 18:15:39 branch-1-router systemd[1]: Starting The Salt Minion... Jun 03 18:15:39 branch-1-router systemd[1]: Started The Salt Minion.

so the salt-minion systemd service is active. Back on the conductor we can see:

[email protected]# show assets Thu 2021-06-03 18:19:04 UTC ============== =============== ===================== ============== ========= ================ ======== Router Node Asset Id 128T Version Status Time in Status Errors ============== =============== ===================== ============== ========= ================ ======== bostonsite1 datacenter1 datacenter-router-1 5.1.0-1.el7 Running 1d 20h 7m 57s 0 dallassite1 branchoffice2 branch-2-router 5.1.0-1.el7 Running 1d 20h 7m 58s 0 nycsite1 conductor1 conductor 5.1.0-1.el7 Running 1d 20h 16m 12s 0 seattlesite1 branchoffice1 branch-1-router 5.1.0-1.el7 Running 2m 22s 0 None None datacenter-router-2 None Pending 1d 20h 17m 55s 0

SUCCESS!

STOP HERE

9. Salt-minion systemd service status is active(running) but conductor asset status is Disconnected

If the output of the ' systemctl status salt-minion ' shell command indicates the service is active, as below:

t128@branch-1-router ~]# sudo systemctl status salt-minion ● salt-minion.service - The Salt Minion Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/salt-minion.service.d └─dnsTimeout.conf, minionWatchdog.conf, restartAlways.conf Active: active (running) since Thu 2021-06-03 17:40:15 UTC; 10min ago Docs: man:salt-minion(1) file:///usr/share/doc/salt/html/contents.html https://docs.saltstack.com/en/latest/contents.html Main PID: 18688 (salt-minion) Tasks: 14 Memory: 76.3M CGroup: /system.slice/salt-minion.service ├─18688 /usr/bin/python3 /usr/bin/salt-minion ├─18732 /usr/bin/python3 /usr/bin/salt-minion └─18734 /usr/bin/python3 /usr/bin/salt-minion Jun 03 17:40:56 branch-1-router salt-minion[18688]: [ERROR ] Exception during resolving address: [Errno 2] Host name lookup failure Jun 03 17:40:56 branch-1-router salt-minion[18688]: [ERROR ] Exception during resolving address: [Errno 2] Host name lookup failure Jun 03 17:41:25 branch-1-router salt-minion[18688]: [WARNING ] The function "module.run" is using its deprecated version and will expire in version "Phosphorus". Jun 03 17:41:25 branch-1-router salt-minion[18688]: [ERROR ] Exception during resolving address: [Errno 2] Host name lookup failure Jun 03 17:41:25 branch-1-router salt-minion[18688]: [ERROR ] Exception during resolving address: [Errno 2] Host name lookup failure Jun 03 17:41:25 branch-1-router salt-minion[18688]: [ERROR ] Exception during resolving address: [Errno 2] Host name lookup failure Jun 03 17:41:27 branch-1-router salt-minion[18688]: 2048 SHA256:jMLuWaTJjpbpWdN/k+V5ysEnz78GDiAW3CRwoRJncUQ branch-1-router (RSA) Jun 03 17:41:27 branch-1-router salt-minion[18688]: [ERROR ] Exception during resolving address: [Errno 2] Host name lookup failure Jun 03 17:41:27 branch-1-router salt-minion[18688]: [ERROR ] Exception during resolving address: [Errno 2] Host name lookup failure Jun 03 17:41:27 branch-1-router salt-minion[18688]: [ERROR ] Exception during resolving address: [Errno 2] Host name lookup failure [root@branch-1-router ~]#

then the problem may lie in the network between the router and conductor. There are several possibilities:

a) Some network paths between the router and conductor nodes may be unreliable
The router nodes salt-minion recieves commands from the conductor's salt-master on the client side of a TCP connection established with conductor TCP port 4505. Each time it returns results it opens a new TCP connection to conductor TCP port 4506. It is possible that as new connections are created, some go through paths which are less reliable, resulting in the asset status of Disconnected. It is likely this condition would manifest itself as an intermittent Disconnected status as only some percentage of new connections would use the problem path(s).

The asset-connection-resiliency feature can be used to mitigate this behavior.

The feature may be enabled at the authority level for all managed routers by configuring the conductor:

[email protected]# configure authority asset-connection-resiliency enabled false true

or at the router level (which overrides the authority level configuration):
conf authority router <router-name> system asset-connection-resiliency enabled

[email protected]# configure authority router seattlesite1 system asset-connection-resiliency enabled false true use-authority-setting

This feature must be used with caution as each salt-minion to salt-master ssh connection consumes additional conductor memory and CPU. Enabling this for all routers managed by a conductor could overwhelm the conductor!

In order to enable this feature, the conductor uses salt to install the 128T-minion-connector rpm on the router. If a new salt-minion -> salt-master connection can never be established, it may not be possible to enable this feature!!!

b) Firewall / routing rules on a router or firewall in between the router node and conductor may be interfering.
Conductor TCP Ports 4505 and 4506 must be reachable by all routers to establish connections.

STOP HERE

10. Inband Management Connection
In this scenario, the salt minion running on the router node / asset connects to the conductor by forwarding traffic to a KNI interface into the 128 data-plane where auto-generated configuration directs traffic to the configured conductor address. The inband connection is mostly generated for you by the conductor, but in order to triage connectivity problems familiarity with what is explicitly configured and what is generated can be helpful. The minimal items which do require configuration are:

a. The conductor address
config authority conductor-address <ip-address>

[email protected]# show config run flat | grep 'conductor-address' ✔ Piping output... config authority conductor-address 4.4.4.3

            b. Network interfaces marked for management connectivity to conductor
           The network interface's conductor property, when set to true, causes the conductor to generate the configuration necessary to send and receive salt messaging via that interface.
           config authority router <route-name> node <node-name> device <device-name> network-interface <network-interface-name> conductor [true|false]

[email protected]# show config run flat | egrep 'network-interface internet.*?conductor' ✔ Piping output... config authority router seattlesite1 node branchoffice1 device-interface wan1 network-interface internet conductor true config authority router bostonsite1 node datacenter1 device-interface wan1 network-interface internet conductor true config authority router dallassite1 node branchoffice2 device-interface wan1 network-interface internet conductor true

This forwards traffic to the conductor at 4.4.4.3 in our example via interface internet. It does not configure using SVR for this traffic. Were SVR used, the generated configuration would have to have been manually edited.

Below is an example of the additional configuration generated.

config authority router seattlesite1 service-route _conductor_1_route_1 name _conductor_1_route_1 config authority router seattlesite1 service-route _conductor_1_route_1 service-name _conductor_1 config authority router seattlesite1 service-route _conductor_1_route_1 generated true config authority router seattlesite1 service-route _conductor_1_route_1 next-hop branchoffice1 internet node-name branchoffice1 config authority router seattlesite1 service-route _conductor_1_route_1 next-hop branchoffice1 internet interface internet config authority router seattlesite1 service-route _conductor_1_route_1 next-hop branchoffice1 internet gateway-ip 1.1.1.1 config authority router seattlesite1 service-route _conductor_1_route_1 reachability-detection enabled false config authority router bostonsite1 service-route _conductor_1_route_1 name _conductor_1_route_1 config authority router bostonsite1 service-route _conductor_1_route_1 service-name _conductor_1 config authority router bostonsite1 service-route _conductor_1_route_1 generated true config authority router bostonsite1 service-route _conductor_1_route_1 next-hop datacenter1 internet node-name datacenter1 config authority router bostonsite1 service-route _conductor_1_route_1 next-hop datacenter1 internet interface internet config authority router bostonsite1 service-route _conductor_1_route_1 next-hop datacenter1 internet gateway-ip 3.3.3.1 config authority router bostonsite1 service-route _conductor_1_route_1 reachability-detection enabled false config authority router dallassite1 service-route _conductor_1_route_1 name _conductor_1_route_1 config authority router dallassite1 service-route _conductor_1_route_1 service-name _conductor_1 config authority router dallassite1 service-route _conductor_1_route_1 generated true config authority router dallassite1 service-route _conductor_1_route_1 next-hop branchoffice2 internet node-name branchoffice2 config authority router dallassite1 service-route _conductor_1_route_1 next-hop branchoffice2 internet interface internet config authority router dallassite1 service-route _conductor_1_route_1 next-hop branchoffice2 internet gateway-ip 2.2.2.1 config authority router dallassite1 service-route _conductor_1_route_1 reachability-detection enabled false config authority service _conductor_1 name _conductor_1 config authority service _conductor_1 enabled true config authority service _conductor_1 scope private config authority service _conductor_1 tap-multiplexing false config authority service _conductor_1 transport icmp protocol icmp config authority service _conductor_1 transport tcp protocol tcp config authority service _conductor_1 transport tcp port-range 443 start-port 443 config authority service _conductor_1 transport tcp port-range 443 end-port 443 config authority service _conductor_1 transport tcp port-range 930 start-port 930 config authority service _conductor_1 transport tcp port-range 930 end-port 930 config authority service _conductor_1 transport tcp port-range 4505 start-port 4505 config authority service _conductor_1 transport tcp port-range 4505 end-port 4505 config authority service _conductor_1 transport tcp port-range 4506 start-port 4506 config authority service _conductor_1 transport tcp port-range 4506 end-port 4506 config authority service _conductor_1 address 4.4.4.3/32 config authority service _conductor_1 generate-categories false config authority service _conductor_1 access-policy-generated true config authority service _conductor_1 access-policy _internal_ source _internal_ config authority service _conductor_1 access-policy _internal_ permission allow config authority service _conductor_1 service-policy _conductor_ config authority service _conductor_1 share-service-routes false config authority service _conductor_1 source-nat network-interface config authority service _conductor_1 application-type generic config authority service _conductor_1 fqdn-resolution-type v4 config authority service _conductor_1 generated true

It is important to note that due to the generated configuration, usually the source IP address of messages sent by the salt-minion will be a 169.254.0.0/16, externally non-routable address. Usually (this may not apply in the case of forwarding via SVR) you will want to have the source-nat property set to true for the network-interface used for forwarding:

show config run flat | egrep 'network-interface <network-interface-name> .*?source-nat'

[email protected]# show config run flat | egrep 'network-interface internet.*?source-nat' ✔ Piping output... config authority router seattlesite1 node branchoffice1 device-interface wan1 network-interface internet source-nat true config authority router bostonsite1 node datacenter1 device-interface wan1 network-interface internet source-nat true config authority router dallassite1 node branchoffice2 device-interface wan1 network-interface internet source-nat true

To see if forwarding rules exist for the conductor at 4.4.4.3 from tenant _internal_ (generated config) to destination service _conductor_1 (also generated), we can look at the FIB (Forwarding Information Base) entries -- either from the conductor, or from the router node with connectivity problems:

[email protected]# show fib router seattlesite1 | egrep 4.4.4.3 ✔ Piping output... 4.4.4.3/32 0 ICMP _internal_ _conductor_1 1.1.1.1 4.4.4.3/32 443 TCP _internal_ _conductor_1 1.1.1.1 4.4.4.3/32 930 TCP _internal_ _conductor_1 1.1.1.1 4.4.4.3/32 4505 TCP _internal_ _conductor_1 1.1.1.1 4.4.4.3/32 4506 TCP _internal_ _conductor_1 1.1.1.1

[email protected]# show fib | egrep 4.4.4.3 ✔ Piping output... 4.4.4.3/32 0 ICMP _internal_ _conductor_1 1.1.1.1 4.4.4.3/32 443 TCP _internal_ _conductor_1 1.1.1.1 4.4.4.3/32 930 TCP _internal_ _conductor_1 1.1.1.1 4.4.4.3/32 4505 TCP _internal_ _conductor_1 1.1.1.1 4.4.4.3/32 4506 TCP _internal_ _conductor_1 1.1.1.1

The last column is the next hop, which in this case is the gateway for the network interface with the conductor property configured as true.

If no entry exists in the FIB for the configured conductor address, then the nexthop gateway may not be reachable or if the configuration was changed recently to use inband management connectivity, perhaps it was misconfigured or did not take.

To check the gateway reachability from the router with conductor connectivity issues, find the gateway from the config. We know that network-interface 'internet' is used to communicate with the conductor, so this regex should extract the info we need from the pcli:

sho config run flat | egrep 'network-interface <network-interface> .*?gateway'

[email protected]# sho config run flat | egrep 'network-interface internet.*?gateway' ✔ Piping output... config authority router seattlesite1 node branchoffice1 device-interface wan1 network-interface internet address 1.1.1.128 gateway 1.1.1.1

now try pinging it:

[email protected]# ping 1.1.1.1 PING 1.1.1.1 56 bytes of data. Ping from 1.1.1.1 (1.1.1.1): icmp_seq=0 ttl=64 time=3.073ms Ping from 1.1.1.1 (1.1.1.1): icmp_seq=1 ttl=64 time=79.775ms Ping from 1.1.1.1 (1.1.1.1): icmp_seq=2 ttl=64 time=90.799ms Ping from 1.1.1.1 (1.1.1.1): icmp_seq=3 ttl=64 time=89.572ms --- 1.1.1.1 ping statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss

If the ping command fails, STOP . you will need to investigate why the gateway is unreachable.

If the ping command succeeds, it is likely the explicit or generated configuration is incorrect. You will need to investigate this, but this is outside the scope of this document.

11. _conductor_1 FIB entries exist, Try Service Ping

At this point, _conductor_1 FIB entries should exist and the next hop should be pingable using the pcli (this feature ias also available in the GUI). The service-ping command can be used to ping the _conductor_1 service to see if there is a reponse. The generated tenant and service should be used along with the configured value for the conductor address:

service-ping tenant _internal service-name _conductor_1 <conductor-address>

[email protected]# service-ping tenant _internal_ service-name _conductor_1 4.4.4.3 PING 4.4.4.3 56 bytes of data. Ping from 4.4.4.3 (4.4.4.3): icmp_seq=0 ttl=63 time=66.948ms Ping from 4.4.4.3 (4.4.4.3): icmp_seq=1 ttl=63 time=17.402ms Ping from 4.4.4.3 (4.4.4.3): icmp_seq=2 ttl=63 time=1.898ms Ping from 4.4.4.3 (4.4.4.3): icmp_seq=3 ttl=63 time=89.977ms --- 4.4.4.3 ping statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 1.898/44.056/89.977/41.314

if the service-ping command fails, STOP and investigate why this might be. Configuration or external network problems are two possibilities. If the management connectivity was configured to be forwarded over SVR, peer reachability needs to be considered and the focus of the investigation might shift to the remote 128 router.

if on the other hand, if the ping command succeeds, but the conductor status for the router node / asset is still Disconnected , the problem may be that source-nat is not configured on the forwarding interface, using the following pcli command on the router:

show config run flat | egrep 'network-interface <network-interface-name> .*?source-nat'

or on the conductor:

show config run flat | egrep 'router <router-name> node <node-name> .*?network-interface <network-interface-name> .*?source-nat'

If the config is missing on both the conductor and router it may need to be added. If it is missing on the router but not the conductor, then try adding it to the router (ignoring the warning about conductor management), and commititng. After waiting the obligitory mintute or 2 w/o the asset status changing from Disconnected to Connected, consider restarting the 128T systemd service on the router from the Linux shell. Restarting the will be service impacting, so only do this if the circumstances permit.

Finally, if none of these steps has helped identify / correct the problem please attach the output of 'save tech-support-info' on both the conductor and router if possible along with the output from the various tests run during this triage session to a trouble ticket.