NA-GRPCD threads pile up in high latency networks eventually leading to a core getting generated on the device.

lab@qfx> show system core-dumps no-forwarding

-rw-rw---- 1 root field  2899458 Mar 28 12:28 /var/tmp/na-grpcd.core.0.gz

Log messages:

Mar  28 12:28:00.013  test_device /kernel: BAD_PAGE_FAULT: pid 57986 (na-grpcd), uid 0: pc 0x88a30db0 got a read fault at 0x500001f0, x86 fault flags = 0x4
Mar  28 12:28:00.013  test_device /kernel: Trapframe Register Dump:
Mar  28 12:28:00.013  test_device /kernel:     eax: 05265c00    ecx: 09c2f000    edx: 09ce0e01    ebx: 50000000
Mar  28 12:28:00.013  test_device /kernel:     esp: 8cdd3f6c    ebp: 8cdd3f98    esi: 00000000    edi: 09c2f088
Mar  28 12:28:00.013  test_device /kernel:     eip: 88a30db0    eflags: 00010202
Mar  28 12:28:00.013  test_device /kernel:     cs: 0033    ss: 003b    ds: 003b    es: 003b
Mar  28 12:28:00.013  test_device /kernel:     fs: b0dc003b    trapno: 0000000c    err: 00000004
Mar  28 12:28:00.013  test_device /kernel: Page table info for PC address 0x88a30db0: PDE = 0x4d702067, PTE = 4ecd8425
Mar  28 12:28:00.013  test_device /kernel: Dumping 16 bytes starting at PC address 0x88a30db0:
Mar  28 12:28:00.013  test_device /kernel:     ff a3 f0 01 00 00 68 c8 03 00 00 e9 50 f8 ff ff 

The na-grpcd core reported would be mostly seen in high latency networks or high latency between DUT and collector. In such high latency cases, there is a bug in the clean up path of terminating subscription threads in na-grpcd.

NA-GRPCD process doesn't terminate the subscription whenever there was a write failure towards jsd, leading to a lot of stale threads. This eventually causes the na-grpcd process to crash. 

The leak of subscription threads was not seen with normal latency in our internal tests, but more profound with higher latencies.

With normal latencies, the connection should get cleared faster and there shouldn't be a pile up of threads in na-grpcd. The rate at which the collectors connect/disconnect also plays a role here. Higher the rate of connect/disconnects more the chances of having the thread leaks with high latency networks. But outside of this, even with a normal network latency, we might see a steady but much smaller increase in the thread count of na-grpcd. This issue may also lead to similar cores if kept for running over a longer period of time. 

This issue is applicable only to freebsd6 based JUNOS platforms and is a day 1 issue.