Telemetry stops working and no data is sent. Each time when subscription occurs, agentd opens a FD to read. The total count of FDs of agentd keeps increasing until SDN-Telemetry process gets restarted. This Issue can be observed on 17.2X75-D103.3 and has been fixed in later releases.
The command 'show agent sensors' does not indicate a cause. The log messages below shows 'Too many open files'.
'show agent sensors'
Verify => procstat -f <agentd.pid> root@re0:/var/home/remote # procstat -f 6819 PID COMM FD T V FLAGS REF OFFSET PRO NAME 6819 agentd text v r r------- - - - /packages/mnt/junos-runtime-mx/usr/sbin/agentd 6819 agentd cwd v d r------- - - - /var/tmp 6819 agentd root v d r------- - - - / 6819 agentd jail v d r------- - - - / 6819 agentd 0 v c rw------ 3 0 - /dev/null 6819 agentd 1 v c rw------ 3 0 - /dev/null 6819 agentd 2 v c rw------ 3 0 - /dev/null 6819 agentd 3 s - rw------ 1 0 UDD /var/run/logpriv 6819 agentd 4 v r rw-----l 68 5 - /var/run/jlaunchd.pid 6819 agentd 5 v r rw-----l 1 5 - /var/run/agentd.pid 6819 agentd 6 s - rw---n-- 1 0 UDS /var/run/agentd_mgmt 6819 agentd 7 p - rw---n-- 1 0 - - 6819 agentd 8 p - rw------ 1 0 - - 6819 agentd 9 v r r------- 1 16 - - 6819 agentd 10 p - rw---n-- 1 0 - - 6819 agentd 11 p - rw------ 1 0 - - 6819 agentd 12 s - rw------ 1 0 UDD /var/run/eventd_events 6819 agentd 13 s - rw---n-- 1 0 ? 6819 agentd 14 v c r------- 1 320 - /dev/random 6819 agentd 15 s - rw---n-- 1 0 ? 6819 agentd 16 s - rw---n-- 1 0 ? 6819 agentd 17 s - rw---n-- 1 0 TCP 0.0.0.0:9500 0.0.0.0:0 6819 agentd 18 s - rw---n-- 1 0 UDP 0.0.0.0:2000 0.0.0.0:0 6819 agentd 19 p - rw---n-- 1 0 - - 6819 agentd 20 p - rw---n-- 1 0 - - 6819 agentd 21 s - rw---n-- 1 0 UDS - 6819 agentd 22 s - rw---n-- 1 0 UDS - 6819 agentd 23 v r r------- 1 16 - - 6819 agentd 24 s - rw---n-- 1 0 TCP 128.0.0.4:54567 128.0.0.4:51883 6819 agentd 25 s - rw---n-- 1 0 TCP 128.0.0.4:9500 128.0.0.4:64870 6819 agentd 26 s - rw---n-- 1 0 UDS /var/run/agentd-26 6819 agentd 29 v r r------- 1 16 - - 6819 agentd 30 v r r------- 1 16 - - 6819 agentd 31 v r r------- 1 16 - - 6819 agentd 32 v r r------- 1 16 - - 6819 agentd 33 v r r------- 1 16 - - 6819 agentd 34 v r r------- 1 16 - - 6819 agentd 35 v r r------- 1 16 - - 6819 agentd 36 v r r------- 1 16 - - 6819 agentd 37 v r r------- 1 16 - - 6819 agentd 38 v r r------- 1 16 - - 6819 agentd 39 v r r------- 1 16 - - 6819 agentd 40 v r r------- 1 16 - - 6819 agentd 41 v r r------- 1 16 - - 6819 agentd 42 v r r------- 1 16 - - 6819 agentd 43 v r r------- 1 16 - - 6819 agentd 44 v r r------- 1 16 - - 6819 agentd 45 v r r------- 1 16 - - 6819 agentd 46 v r r------- 1 16 - - 6819 agentd 47 v r r------- 1 16 - - 6819 agentd 48 v r r------- 1 16 - - 6819 agentd 49 v r r------- 1 16 - - 6819 agentd 50 v r r------- 1 16 - - 6819 agentd 51 v r r------- 1 16 - - 6819 agentd 52 v r r------- 1 16 - - 6819 agentd 53 v r r------- 1 16 - - 6819 agentd 54 v r r------- 1 16 - - 6819 agentd 55 v r r------- 1 16 - - 6819 agentd 56 v r r------- 1 16 - - 6819 agentd 57 v r r------- 1 16 - - 6819 agentd 58 v r r------- 1 16 - - 6819 agentd 59 v r r------- 1 16 - - 6819 agentd 60 v r r------- 1 16 - - 6819 agentd 61 v r r------- 1 16 - - 6819 agentd 62 v r r------- 1 16 - - 6819 agentd 63 v r r------- 1 16 - - 6819 agentd 64 v r r------- 1 16 - - 6819 agentd 65 v r r------- 1 16 - - 6819 agentd 66 v r r------- 1 16 - - 6819 agentd 67 v r r------- 1 16 - - 6819 agentd 68 v r r------- 1 16 - - 6819 agentd 69 v r r------- 1 16 - - 6819 agentd 70 v r r------- 1 16 - - 6819 agentd 71 v r r------- 1 16 - - 6819 agentd 72 v r r------- 1 16 - - 6819 agentd 73 v r r------- 1 16 - - 6819 agentd 74 v r r------- 1 16 - - 6819 agentd 75 v r r------- 1 16 - - 6819 agentd 76 v r r------- 1 16 - - 6819 agentd 77 v r r------- 1 16 - - 6819 agentd 78 v r r------- 1 16 - - 6819 agentd 79 v r r------- 1 16 - - 6819 agentd 80 v r r------- 1 16 - - 6819 agentd 81 v r r------- 1 16 - - 6819 agentd 82 v r r------- 1 16 - - 6819 agentd 83 v r r------- 1 16 - - 6819 agentd 84 v r r------- 1 16 - - 6819 agentd 85 v r r------- 1 16 - - 6819 agentd 86 v r r------- 1 16 - - 6819 agentd 87 v r r------- 1 16 - - 6819 agentd 88 v r r------- 1 16 - - 6819 agentd 89 v r r------- 1 16 - - 6819 agentd 90 v r r------- 1 16 - - 6819 agentd 91 v r r------- 1 16 - - 6819 agentd 92 v r r------- 1 16 - -
Log:
Feb 24 05:37:43 re0 mosquitto[6710]: Socket error on client client-1, disconnecting. Feb 24 05:37:35 re0 mgd[84106]: UI_COMMIT_PROGRESS: Commit operation in progress: notifying none(0) Feb 24 05:37:35 re0 mgd[84106]: UI_COMMIT_PROGRESS: Commit operation in progress: cannot signal 'None process', signal 1, no pid Feb 24 05:37:35 re0 mgd[84106]: UI_COMMIT_PROGRESS: Commit operation in progress: notifying agentd(18) Feb 24 05:37:35 re0 mgd[84106]: UI_COMMIT_PROGRESS: Commit operation in progress: signaling 'agent for all the telemetry sensors', pid 6720, signal 1, status 0 with notification errors enabled Feb 24 05:37:35 re0 agentd[6720]: agentd_sighup_proc: AgentD received SIGHUP. is_master:1 Feb 24 05:37:35 re0 mgd[84106]: UI_EPHEMERAL_COMMIT_COMPLETED: commit complete on 'junos-analytics' ephemeral database Feb 24 05:37:35 re0 agentd[6720]: UI_DBASE_OPEN_FAILED: Database open failed for file '/var/run/db/juniper.eph_junos-analytics': Too many open files Feb 24 05:37:35 re0 agentd[6720]: agentd_config_read_traceoptions 94: /var/log/agentd /var/log/agentd 0 0 0 Feb 24 05:37:35 re0 agentd[6720]: agentd_config_read: Entering config read. is_master:1 Feb 24 05:37:35 re0 mgd-api[6510]: MGD_API_APPLICATION_DISCONNECT: Client disconnected from mgd-api Feb 24 05:37:43 re0 sshd[84115]: Did not receive identification string from 10.170.248.68 Feb 24 05:37:43 re0 inetd[6456]: /usr/sbin/sshd[84115]: exited, status 255 Feb 24 05:37:43 re0 mosquitto[6710]: Socket error on client client-1, disconnecting.
Each time when subscription occurs, agentd opens the FD to read and max open FD total count is 1024. There was an FD leak and when FD reaches to 1024, agentd stops reading the configuration.
Restarting SDN-telemetry will resolve the problem. However, this issue has been fixed in releases after 17.2X75-D103.3.