Article Detail

Description

Any site that is provisioned through Contrail Service Orchestration will establish outbound ssh connections to the CSO. Device connectivity microservice part of CSO will maintain connections initiated by all sites in zookeeper database.

Solution

Based on the device type and component , CSO will store respective device connections information in the zookeeper. Device connection types stored can be of two types

DIC (Device Initiated Connection): This will be outbound TCP connection originated by respective device component. There should be "system services outbound-ssh client" configuration in the respective device component.
TT (Tunnel Through) : These type of connections are on-demand and may be created when user attempts to deploy configurations to the device through CSO. These are logical connection ( port forwarding) types which will rely on DIC type connections to connect with respective component. This type of connections will be typically created for virtual components like JCP and GWR in NFX devices where communications between CSO and JCP/GWR has to go via JDM device connection.

As mentioned in KB37298 [juniper.net] operator who has access to infra can login to zookeeper Kubernetes PODS to access device connection related information. CSO Device connectivity (aka DCS - Device Connectivity service) microservice maintains various connections based on the Device flavors. Following are more details based on the device flavor.

Dual NFX250 [ SPOKE ] :

Here is the sample Dual NFX250 device screenshot in UI. In the screenshot it also shows the UUID for Cluster, CPE0 and CPE1 device objects. DCS will store the device connections using UUID as prefix as shown below.

:: Zookeeper cli Outputs ::

Following are the list of znodes (zookeeper nodes) corresponding to device connections maintained for Dual NFX250 site that was shown in above example.

[zk: localhost:2181(CONNECTED) 2] ls /device_connectivity/devices
[e2116d58-d5bc-4a79-997c-a96028ee59a4:GWR, b00d11ea-41d3-4bc8-a515-36208704dcf2:JDM, b00d11ea-41d3-4bc8-a515-36208704dcf2:JCP, f169533d-a530-4464-a952-7102b257a1df:JDM, b00d11ea-41d3-4bc8-a515-36208704dcf2:GWR, f169533d-a530-4464-a952-7102b257a1df:JCP, f169533d-a530-4464-a952-7102b257a1df:GWR]

Here are additional details of each znode. It is very important that GWR/JCP TT connection timestamp for respective NFX member nodes should be always greater than JDM DIC connection. Otherwise this will result in mismatch and DCS microservice assumes device has no connection and eventually result in job failure.

[zk: localhost:2181(CONNECTED) 4] get /device_connectivity/devices/e2116d58-d5bc-4a79-997c-a96028ee59a4:GWR
{"ip": "GWR.DUAL-NFX33-34.EastCoast", "conntype": "DIC", "serviceid": "csp.csp-device-connectivity-7f97d486dc-gpb8s", "srcport": 63804, "starttime": 1683737598}
cZxid = 0x40155cdd8
ctime = Wed May 10 16:53:18 GMT 2023
mZxid = 0x40155cdd8
mtime = Wed May 10 16:53:18 GMT 2023
pZxid = 0x40155cdd8
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 160
numChildren = 0

[zk: localhost:2181(CONNECTED) 5]
[zk: localhost:2181(CONNECTED) 5] get /device_connectivity/devices/f169533d-a530-4464-a952-7102b257a1df:JDM
{"ip": null, "conntype": "DIC", "serviceid": "csp.csp-device-connectivity-7f97d486dc-q7sp6", "srcport": 25014, "starttime": 1683737599}
cZxid = 0x40155ce00
ctime = Wed May 10 16:53:19 GMT 2023
mZxid = 0x40155ce00
mtime = Wed May 10 16:53:19 GMT 2023
pZxid = 0x40155ce00
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 135
numChildren = 0

[zk: localhost:2181(CONNECTED) 3] get /device_connectivity/devices/f169533d-a530-4464-a952-7102b257a1df:JCP
{"ip": "vjunos0", "conntype": "TT", "serviceid": "csp.csp-device-connectivity-7f97d486dc-q7sp6", "srcport": 25014, "starttime": 1683846527}
cZxid = 0x401731690
ctime = Thu May 11 23:08:47 GMT 2023
mZxid = 0x401731690
mtime = Thu May 11 23:08:47 GMT 2023
pZxid = 0x401731690
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 139
numChildren = 0

[zk: localhost:2181(CONNECTED) 4] get /device_connectivity/devices/f169533d-a530-4464-a952-7102b257a1df:GWR
{"ip": "GWR.DUAL-NFX33-34.EastCoast", "conntype": "TT", "serviceid": "csp.csp-device-connectivity-7f97d486dc-q7sp6", "srcport": 25014, "starttime": 1683846513}
cZxid = 0x401731565
ctime = Thu May 11 23:08:33 GMT 2023
mZxid = 0x401731565
mtime = Thu May 11 23:08:33 GMT 2023
pZxid = 0x401731565
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 159
numChildren = 0

[zk: localhost:2181(CONNECTED) 6] get /device_connectivity/devices/b00d11ea-41d3-4bc8-a515-36208704dcf2:JCP
{"ip": "vjunos0", "conntype": "TT", "serviceid": "csp.csp-device-connectivity-7f97d486dc-4fj75", "srcport": 30717, "starttime": 1683846526}
cZxid = 0x40173167b
ctime = Thu May 11 23:08:46 GMT 2023
mZxid = 0x40173167b
mtime = Thu May 11 23:08:46 GMT 2023
pZxid = 0x40173167b
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 139
numChildren = 0

[zk: localhost:2181(CONNECTED) 5] get /device_connectivity/devices/b00d11ea-41d3-4bc8-a515-36208704dcf2:GWR
{"ip": "GWR.DUAL-NFX33-34.EastCoast", "conntype": "TT", "serviceid": "csp.csp-device-connectivity-7f97d486dc-4fj75", "srcport": 30717, "starttime": 1683846513}
cZxid = 0x40173155c
ctime = Thu May 11 23:08:33 GMT 2023
mZxid = 0x40173155c
mtime = Thu May 11 23:08:33 GMT 2023
pZxid = 0x40173155c
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 159
numChildren = 0

Here is the summary of connections details for Dual NFX250.

Node	Component	Connection Type
CPE0	JDM	DIC
CPE0	JCP	TT
CPE0	GWR	TT
CPE1	JDM	DIC
CPE1	JCP	TT
CPE1	GWR	TT
CLUSTER	GWR	DIC

% CPE 0 %

f169533d-a530-4464-a952-7102b257a1df:JDM [ DIC connection ]

f169533d-a530-4464-a952-7102b257a1df:GWR [ TT connection ]

f169533d-a530-4464-a952-7102b257a1df: JCP [ TT connection ]

% CPE 1 %

b00d11ea-41d3-4bc8-a515-36208704dcf2:JDM   [ DIC connection]

b00d11ea-41d3-4bc8-a515-36208704dcf2:GWR   [ TT connection ]

b00d11ea-41d3-4bc8-a515-36208704dcf2:JCP [ TT connection ]

% Cluster %

e2116d58-d5bc-4a79-997c-a96028ee59a4:GWR [ DIC connection ]

:: Junos CLI OUTPUTS ::

Correlating the Zookeeper device connection with CLI outputs . As mentioned earlier all DIC connection types will have respective outbound-ssh client configurations.

% CPE-0 JDM %

Outbound ssh configuration snippet from Node 0 JDM.

root@jdm> show configuration | display set | match outbound
set system services outbound-ssh client CSO-c8962c88-ef4e-11ed-b0af-62f72150e20c device-id f169533d-a530-4464-a952-7102b257a1df.JDM
set system services outbound-ssh client CSO-c8962c88-ef4e-11ed-b0af-62f72150e20c keep-alive
set system services outbound-ssh client CSO-c8962c88-ef4e-11ed-b0af-62f72150e20c services netconf
set system services outbound-ssh client CSO-c8962c88-ef4e-11ed-b0af-62f72150e20c 10.85.189.66 port 7804

jdm:~# netstat -antup | grep 7804
tcp        0      0 100.123.250.1:46668     10.85.189.66:7804       ESTABLISHED 6341/sshd: csp [pri
jdm:~#

jdm:~# netstat -antup | grep 192.0.2.100
tcp        0      0 192.0.2.254:37565       192.0.2.100:22          ESTABLISHED 6345/sshd: csp@nott.    <<<<<<< This is the port forwarding connection TT connection type relies on to communicate with GWR from JDM.

jdm:~# netstat -antup | grep 192.0.2.2
tcp        0      0 192.0.2.254:37202       192.0.2.2:22            ESTABLISHED 6345/sshd: csp@nott. <<<<< This is the port forwarding connection TT connection type relies on to communicate with JCP from JDM.

% CPE-1 JDM %

Outbound ssh configuration snippet from Node 1 JDM.
root@jdm> show configuration | display set | match outbound
set system services outbound-ssh client CSO-c8a6da24-ef4e-11ed-b0af-62f72150e20c device-id b00d11ea-41d3-4bc8-a515-36208704dcf2.JDM
set system services outbound-ssh client CSO-c8a6da24-ef4e-11ed-b0af-62f72150e20c keep-alive
set system services outbound-ssh client CSO-c8a6da24-ef4e-11ed-b0af-62f72150e20c services netconf
set system services outbound-ssh client CSO-c8a6da24-ef4e-11ed-b0af-62f72150e20c 10.85.189.66 port 7804

jdm:~# netstat -antup | grep 7804
tcp        0      0 100.123.250.1:46668     10.85.189.66:7804       ESTABLISHED 6341/sshd: csp [pri
jdm:~#

jdm:~# netstat -antup | grep 192.0.2.100
tcp        0      0 192.0.2.254:45733       192.0.2.100:22          ESTABLISHED 5559/sshd: csp@nott   <<<<<<< This is the port forwarding connection TT connection type relies on to communicate with GWR from JDM.

jdm:~# netstat -antup | grep 192.0.2.2
tcp        0      0 192.0.2.254:40734       192.0.2.2:22            ESTABLISHED 5559/sshd: csp@nott <<<<< This is the port forwarding connection TT connection type relies on to communicate with JCP from JDM

Note: CSO use username 'csp' to securely connect to devices using ssh keys.

% GWR / vSRX cluster CLI %

Since GWR is the only way for traffic to leave the NFX device , there will be 3 outbound ssh connections from Dual NFX.

10.64.30.48 --> lo0 on vSRX cluster
100.123.250.1 --> jsxe0.4010 on JDM CPE0
100.123.250.4 --> jsxe0.4010 on JDM CPE1

{primary:node0}
root@NFX> show security flow session destination-port 7804
node0:
--------------------------------------------------------------------------

Session ID: 110, Policy name: self-traffic-policy/1, State: Active, Timeout: 1792, Valid
In: 10.64.30.48/62450 --> 10.85.189.66/7804;tcp, Conn Tag: 0x0, If: .local..0, Pkts: 23936, Bytes: 4874072,
Out: 10.85.189.66/7804 --> 10.64.30.48/32260;tcp, Conn Tag: 0x0, If: st0.4005, Pkts: 12026, Bytes: 881897,

Session ID: 111, Policy name: default-permit/6, State: Active, Timeout: 1790, Valid
In: 100.123.250.1/46668 --> 10.85.189.66/7804;tcp, Conn Tag: 0x0, If: reth7.0, Pkts: 1277, Bytes: 183743,
Out: 10.85.189.66/7804 --> 10.64.30.48/32261;tcp, Conn Tag: 0x0, If: st0.4005, Pkts: 668, Bytes: 91937,

Session ID: 142, Policy name: default-permit/6, State: Active, Timeout: 1798, Valid
In: 100.123.250.4/49929 --> 10.85.189.66/7804;tcp, Conn Tag: 0x0, If: reth7.0, Pkts: 1272, Bytes: 183179,
Out: 10.85.189.66/7804 --> 10.64.30.48/32263;tcp, Conn Tag: 0x0, If: st0.4001, Pkts: 668, Bytes: 91777,
Total sessions: 3

node1:
--------------------------------------------------------------------------

Session ID: 37, Policy name: self-traffic-policy/1, State: Backup, Timeout: 5850, Valid
In: 10.64.30.48/62450 --> 10.85.189.66/7804;tcp, Conn Tag: 0x0, If: .local..0, Pkts: 0, Bytes: 0,
Out: 10.85.189.66/7804 --> 10.64.30.48/32260;tcp, Conn Tag: 0x0, If: st0.4005, Pkts: 0, Bytes: 0,

Session ID: 38, Policy name: default-permit/6, State: Backup, Timeout: 5864, Valid
In: 100.123.250.1/46668 --> 10.85.189.66/7804;tcp, Conn Tag: 0x0, If: reth7.0, Pkts: 0, Bytes: 0,
Out: 10.85.189.66/7804 --> 10.64.30.48/32261;tcp, Conn Tag: 0x0, If: st0.4005, Pkts: 0, Bytes: 0,

Session ID: 56, Policy name: default-permit/6, State: Backup, Timeout: 5910, Valid
In: 100.123.250.4/49929 --> 10.85.189.66/7804;tcp, Conn Tag: 0x0, If: reth7.0, Pkts: 0, Bytes: 0,
Out: 10.85.189.66/7804 --> 10.64.30.48/32263;tcp, Conn Tag: 0x0, If: st0.4001, Pkts: 0, Bytes: 0,
Total sessions: 3

SRX4200 [ PHUB ] :

Just like Spoke Device, for PHUB which is SRX4200 type will have following device connection information stored in CSO.

Here is the sample Provide Hub devices device screenshot in UI. In the screenshot it also shows the UUID for the PHUB devices. Following example will use "HUBSRX-r013" as an example.

:: Zookeeper cli Outputs ::

[zk: localhost:2181(CONNECTED) 2] ls /device_connectivity/devices

[ 596a4bcb-652b-4d9d-8dca-2f2398d4efef:CLI, 596a4bcb-652b-4d9d-8dca-2f2398d4efef:JUNOS ]

[zk: localhost:2181(CONNECTED) 10] get /device_connectivity/devices/596a4bcb-652b-4d9d-8dca-2f2398d4efef:CLI
{"ip": null, "conntype": "DIC", "serviceid": "csp.csp-device-connectivity-7f97d486dc-vvtx5", "srcport": "60541", "starttime": 1682466300}
cZxid = 0x4000003a8
ctime = Tue Apr 25 23:45:00 GMT 2023
mZxid = 0x4000003a8
mtime = Tue Apr 25 23:45:00 GMT 2023
pZxid = 0x4000003a8
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 137
numChildren = 0
[zk: localhost:2181(CONNECTED) 11] get /device_connectivity/devices/596a4bcb-652b-4d9d-8dca-2f2398d4efef:JUNOS
{"ip": null, "conntype": "DIC", "serviceid": "csp.csp-device-connectivity-7f97d486dc-vvtx5", "srcport": 58208, "starttime": 1682466298}
cZxid = 0x400000325
ctime = Tue Apr 25 23:44:58 GMT 2023
mZxid = 0x400000325
mtime = Tue Apr 25 23:44:58 GMT 2023
pZxid = 0x400000325
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 135
numChildren = 0

Here is the summary of connections details for standalone SRX4200 HUB device.

Node	Component	Connection Type
SRX4200	Junos	DIC
SRX4200	CLI	DIC

% SRX4200 %

596a4bcb-652b-4d9d-8dca-2f2398d4efef:CLI [DIC connection]
596a4bcb-652b-4d9d-8dca-2f2398d4efef:JUNOS. [DIC connection ]

:: Junos CLI OUTPUTS ::

Correlating the Zookeeper device connection with CLI outputs . As mentioned earlier all DIC connection types will have respective outbound-ssh client configurations.

% SRX4200 HUB%

root@HUBSRX> show configuration | display set | match CSO-
set system services outbound-ssh client CSO-78f02f8e-44ff-11ed-878e-fa50e4dedb81 device-id 596a4bcb-652b-4d9d-8dca-2f2398d4efef.JUNOS
set system services outbound-ssh client CSO-78f02f8e-44ff-11ed-878e-fa50e4dedb81 keep-alive
set system services outbound-ssh client CSO-78f02f8e-44ff-11ed-878e-fa50e4dedb81 services netconf
set system services outbound-ssh client CSO-78f02f8e-44ff-11ed-878e-fa50e4dedb81 10.85.189.66 port 7804

root@HUBSRX> show configuration | display set | match CSO_CLI
set system services outbound-ssh client CSO_CLI_TERMINAL device-id 596a4bcb-652b-4d9d-8dca-2f2398d4efef.CLI
set system services outbound-ssh client CSO_CLI_TERMINAL keep-alive
set system services outbound-ssh client CSO_CLI_TERMINAL services netconf
set system services outbound-ssh client CSO_CLI_TERMINAL 10.85.189.66 port 7804

Modification History

re publishing

Description

Solution

Dual NFX250 [ SPOKE ] :

SRX4200 [ PHUB ] :

Modification History