Description

Any site that is provisioned through Contrail Service Orchestration will establish outbound ssh connections to the CSO. Device connectivity microservice part of CSO will maintain connections initiated by all sites in zookeeper database. This article explains how to verify connections between DCS and zookeeper  

Symptoms

Contrail Service Orchestration installed in on-prem or in SaaS in High availability mode.

Solution


Here are the steps to verify the connectivity between DCS microservice and Zookeeper in CSO HA deployment. 
  • As mentioned in KB37289 [juniper.net]  , in on-prem HA deployment user can login to the respective K8 microservice node. In microservice worker node identify the DCS container id. 
 
root@k8-microservices3:~# docker ps | grep device
97be9461f5b0        1bbceb2e0c40                                                    "/csp-service-docker…"   30 minutes ago      Up 30 minutes                           k8s_device-conn_csp.csp-device-connectivity-7f97d486dc-d2tv2_regional_dbee3ac1-86b9-4f9d-bbf9-58c795a36bcd_0
6c6a6ee223ad        csp-installer:10000/csp-docker/pause:3.1                        "/pause"                 30 minutes ago      Up 30 minutes                           k8s_POD_csp.csp-device-connectivity-7f97d486dc-d2tv2_regional_dbee3ac1-86b9-4f9d-bbf9-58c795a36bcd_0
f19db4d95842        1bbceb2e0c40                                                    "/csp-service-docker…"   32 minutes ago      Up 32 minutes                           k8s_device-conn_csp.csp-device-connectivity-7f97d486dc-knhc2_regional_ac2f27cb-7dd1-4493-8108-938b6f30c4c2_0
72445a3f934d        csp-installer:10000/csp-docker/pause:3.1                        "/pause"                 32 minutes ago      Up 32 minutes                           k8s_POD_csp.csp-device-connectivity-7f97d486dc-knhc2_regional_ac2f27cb-7dd1-4493-8108-938b6f30c4c2_0
8efb390d1d9b        1bbceb2e0c40                                                    "/csp-service-docker…"   17 hours ago        Up 17 hours                             k8s_device-conn_csp.csp-device-connectivity-7f97d486dc-4fj75_regional_da3178f7-26ab-40ef-aea7-e07bab1bf5f5_0
d9c02fd55047        csp-installer:10000/csp-docker/pause:3.1                        "/pause"                 17 hours ago        Up 17 hours                             k8s_POD_csp.csp-device-connectivity-7f97d486dc-4fj75_regional_da3178f7-26ab-40ef-aea7-e07bab1bf5f5_0
 
  • Get the PID for a specific DCS container. 
root@k8-microservices3:~# docker inspect -f '{{.State.Pid}}' 97be9461f5b0
32413
root@k8-microservices3:~#
root@k8-microservices3:~#
  • Zookeeper service will listen on tcp port 2181.Verify the tcp connections with destination port 2181 for that specific container. . Here there are 3 sessions that load balanced to 3 zookeeper instances due to HA. 
root@k8-microservices3:~# nsenter -t 32413 -n netstat -antup | grep 2181
tcp        0      0 10.244.44.11:58172      10.104.112.251:2181     ESTABLISHED 32478/python
tcp        0      0 10.244.44.11:58258      10.104.112.251:2181     ESTABLISHED 32476/python
tcp        0      0 10.244.44.11:58290      10.104.112.251:2181     ESTABLISHED 32471/python
root@k8-microservices3:~#
root@k8-microservices3:~# tcpd
tcpd       tcpdchk    tcpdmatch  tcpdump
root@k8-microservices3:~# ps -ef | grep 32478
root     11655  3474  0 16:41 pts/0    00:00:00 grep --color=auto 32478
999      32478 32413  3 16:04 ?        00:01:16 /usr/bin/python /usr/local/bin/launch_dcs_cmdexecuter --config-file /etc/csp-service/application.conf --config-file /etc/csp-service/keystone-auth.conf --logging_conf /etc/csp-service/logging.conf
 
 
root@startupserver1:~# kubectl get pods -n infra | grep zoo
zookeeper-zookeeper-0                                      1/1     Running   0          202d
zookeeper-zookeeper-1                                      1/1     Running   0          202d
zookeeper-zookeeper-2                                      1/1     Running   0          202d
 
 

Modification History

draft