On QFX5100, streaming telemetry clients were keep getting disconnected. The grpc connection uses TCP which gets stuck in FIN_WAIT_2. The pings work fine but the clients are not receiving any stats from the device.

Troubleshooting steps taken: Verified device configs, verified that streaming telemetry server is reachable through ping, checked TCP connection and determined that the device seems to be trying to end the connection with the telemetry server.

Pings to the system are successful but the TCP connection shows that it is in FIN_WAIT_2 state instead of ESTABLISHED. 

The switch can ping the clients.

user@switch> ping 2002:::1

PING6(56=40+8+8 bytes) 2a00::2 --> 2002::1

16 bytes from 2002::1, icmp_seq=0 hlim=118 time=334.945 ms
16 bytes from 2002::1, icmp_seq=1 hlim=118 time=333.341 ms
16 bytes from 2002::1 icmp_seq=2 hlim=118 time=11.239 ms
16 bytes from 2002::1, icmp_seq=3 hlim=118 time=11.218 ms

--- 2002::1 ping6 statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 11.218/172.686/334.945/161.458 ms

However, the TCP connection could not stay in established state.

user@switch> start shell
% netstat -an | grep 10162

tcp6 0 0 2a00::1.10162 2002::1.53350 FIN_WAIT_2
tcp6 0 0 2a00::1.10162 2002::2.54676 FIN_WAIT_2

We enabled the traceoptions through the following config, however since the TCP connection never stayed stable, there traces didn't help much.

set system services extension-service traceoptions file grpc-trace
set system services extension-service traceoptions file size 10m
set system services extension-service traceoptions file files 10
set system services extension-service traceoptions flag all
set system services extension-service traceoptions flag libgrpc-debug

Following logs can be seen in the grpc-trace,

Nov  4 08:02:47 Listen: Grpc channel to unix:/var/run/japi_na-grpcd transitioned from GRPC_CHANNEL_IDLE state to GRPC_CHANNEL_CONNECTING for client ipv6:2002::7:33446 
Nov  4 08:02:47 Listen: Grpc channel to unix:/var/run/japi_na-grpcd transitioned from GRPC_CHANNEL_CONNECTING state to GRPC_CHANNEL_TRANSIENT_FAILURE, respective clients will be terminated
Nov  4 08:02:47 DeleteChannel: Deleting entries in tag map for client ipv6:2002::7:33446
Nov  4 08:02:47 DeleteChannel: Session for Client ipv6:2002::7:33446 will be terminated as a channel transitioned to FATAL/TRANSIENT FAILURE
Nov  4 08:02:47 TerminateClient: Client ipv6:2002::7:33446 will be terminated

We noticed that na-grpcd logs did not have any recent logs written to it. "show system processes extensive" shows na-grpcd process was also not running.

Restarting the na-grpcd process fixed the issue.

user@switch> show system processes extensive | match na-grpcd

user@switch> restart na-grpc-server

error: Daemon providing gRPC interfaces is not running
Daemon providing gRPC interfaces started, pid 40949


user@switch> show system processes extensive | match na-grpcd

40949 root 40 0 19056K 9004K select 0:00 0.00% na-grpcd
40949 root 40 0 19056K 9004K select 0:00 0.00% na-grpcd
40949 root 40 0 19056K 9004K select 0:00 0.00% na-grpcd
40949 root 40 0 19056K 9004K select 0:00 0.00% na-grpcd
40949 root 40 0 19056K 9004K select 0:00 0.00% na-grpcd
40949 root 40 0 19056K 9004K ucond 0:00 0.00% na-grpcd
40949 root 8 0 19056K 9004K nanslp 0:00 0.00% na-grpcd