Switch reporting high MGD process , "routing subsystem is not running" and jsd core dump.
The switch will report low memory condition causing routing-subsystem to fail and jsd core dump.
root@Switch> show route error: the routing subsystem is not running root@Switch> show system processes extensive no-forwarding last pid: 20055; load averages: 1.30, 0.85, 0.43 up 0+00:08:53 08:33:04 468 threads: 6 running, 424 sleeping, 38 waiting CPU: 13.7% user, 0.0% nice, 6.7% system, 0.6% interrupt, 79.0% idle Mem: 1241M Active, 1561M Inact, 325M Wired, 76M Buf, 690M Free Swap: 3072M Total, 3072M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 16994 root 103 0 508M 77M CPU3 3 3:24 100.00% mgd 11 root 155 ki31 0B 64K RUN 0 7:09 76.17% idle{idle: cpu0} 11 root 155 ki31 0B 64K RUN 3 7:07 75.78% idle{idle: cpu3} 11 root 155 ki31 0B 64K CPU2 2 6:52 74.66% idle{idle: cpu2} 11 root 155 ki31 0B 64K RUN 1 7:01 68.65% idle{idle: cpu1} root@Switch> show system core-dumps no-forwarding -rwxrwxrwx 1 root wheel 2686504 Oct 28 08:16 /var/tmp/jsd.core-tarball.0.tgz* >>>>>>> -rwxrwxrwx 1 root wheel 9301210 Sep 18 02:34 /var/tmp/mgd.core.0.gz* -rwxrwxrwx 1 root wheel 4018251 Oct 25 22:03 /var/tmp/na-grpcd.core-tarball.0.tgz* total files: 3 Host Crash Info: ------------------------------------------------------ No corefiles present on the host.
Log messages: Oct 27 07:45:17 Switch kernel: rts_veto_sys_check_low_on_pages: Severe memory pressure, low-free-pages is below threshold. Op 4 err 55, rtsm_id 0:-1, msg type 7, veto simulation: 0. Oct 27 07:46:40 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_RECOVERED: System recovered from low memory condition Oct 27 07:48:49 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_RECOVERED: System recovered from low memory condition Oct 27 07:50:05 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_MINOR_EVENT: System under low memory condition Oct 27 07:50:10 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_RECOVERED: System recovered from low memory condition Oct 27 07:52:15 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_RECOVERED: System recovered from low memory condition Oct 27 07:53:20 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_MINOR_EVENT: System under low memory condition Oct 27 07:53:25 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_RECOVERED: System recovered from low memory condition Oct 27 07:55:36 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_RECOVERED: System recovered from low memory condition Oct 27 07:56:42 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_MINOR_EVENT: System under low memory condition Oct 27 07:56:47 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_MINOR_EVENT: System under low memory condition Oct 27 07:56:52 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_RECOVERED: System recovered from low memory condition Oct 27 07:57:58 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_MINOR_EVENT: System under low memory condition Oct 27 07:58:03 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_RECOVERED: System recovered from low memory condition Oct 27 07:59:08 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_RECOVERED: System recovered from low memory condition Oct 28 08:13:10 Switch last message repeated 5 times Oct 28 08:13:12 Switch smid: SMID_MFS_CAPACITY_ERROR: /mfs capacity 91% Oct 28 08:13:15 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_MINOR_EVENT: System under low memory condition Oct 28 08:13:16 Switch jdhcpd: DH_SVC_SENDMSG_FAILURE: sendmsg() from 0.0.0.0 to port 67 at 255.255.255.255 via interface 7 and routing instance default failed: Network is down Oct 28 08:13:20 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_MINOR_EVENT: System under low memory condition Oct 28 08:13:30 Switch last message repeated 2 times Oct 28 08:13:34 Switch na-grpcd[95431]: NA_GRPCD_CONFIG_EDIT_FAILURE: Ephemeral DB Edit config: error_code=4, error_message='Deadline Exceeded' Oct 28 08:13:35 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_MINOR_EVENT: System under low memory condition Oct 28 08:13:40 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_MINOR_EVENT: System under low memory condition Oct 28 08:13:42 Switch smid: SMID_MFS_CAPACITY_ERROR: /mfs capacity 98% Oct 28 08:13:45 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_MINOR_EVENT: System under low memory condition Oct 28 08:13:55 Switch last message repeated 2 times Oct 28 08:13:55 Switch mgd[1404]: UI_COMMIT: User 'mist' requested 'commit' operation (comment: none) Oct 28 08:13:55 Switch mgd[1404]: UI_EPHEMERAL_COMMIT: User 'mist' has requested commit on 'junos-analytics' ephemeral database Oct 28 08:14:00 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_MINOR_EVENT: System under low memory condition Oct 28 08:14:10 Switch last message repeated 2 times Oct 28 08:14:15 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_MINOR_EVENT: System under low memory condition Oct 28 08:14:16 Switch vccpd[13423]: JTASK_SCHED_SLIP_KEVENT: 14 sec 889047 usec kevent block Oct 28 08:14:16 Switch dot1xd[14773]: JTASK_SCHED_SLIP_KEVENT: 21 sec 515153 usec kevent block Oct 28 08:14:16 Switch mplsoamd[14721]: JTASK_SCHED_SLIP_KEVENT: 19 sec 396481 usec kevent block Oct 28 08:14:20 Switch jdhcpd: DH_SVC_SENDMSG_FAILURE: sendmsg() from 0.0.0.0 to port 67 at 255.255.255.255 via interface 7 and routing instance default failed: Network is down Oct 28 08:14:20 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_MINOR_EVENT: System under low memory condition Oct 28 08:14:35 Switch last message repeated 3 times Oct 28 08:14:35 Switch na-grpcd[95431]: NA_GRPCD_CONFIG_EDIT_FAILURE: Ephemeral DB Edit config: error_code=4, error_message='Deadline Exceeded' Oct 28 08:14:40 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_MINOR_EVENT: System under low memory condition Oct 28 08:14:45 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_MINOR_EVENT: System under low memory condition Oct 28 08:14:45 Switch jlaunchd[13066]: jsd (PID 92593) terminated by signal number 11. Core dumped! Oct 28 08:14:45 Switch jlaunchd[13066]: Registered PID 1460((null)): exec_command Oct 28 08:14:45 Switch jlaunchd[13066]: Dump Command: /bin/sh (PID 1460) started Oct 28 08:14:45 Switch jlaunchd[13066]: Registered PID 1461(jsd): exec_command Oct 28 08:14:45 Switch jlaunchd[13066]: jsd (PID 1461) started Oct 28 08:14:45 Switch jlaunchd[13066]: Registered PID 1461(jsd): new process Oct 28 08:14:47 Switch kernel: rts_veto_sys_check_low_on_pages: Severe memory pressure, low-free-pages is below threshold. Op 4 err 55, rtsm_id 0:-1, msg type 72, veto simulation: 0. Oct 28 08:14:47 Switch kernel: rts_veto_sys_check_low_on_pages: number of pages free = (3685) free page threshold = (3692) number of free pages desired = (4768) veto threshold (high) = (4230) veto threshold (low) = (3692) error = No Buffer curproc = ifinfo. Oct 28 08:14:47 Switch kernel: pid 14763 (jdhcpd), jid 0, uid 0, was killed: out of swap space Oct 28 08:14:47 Switch jlaunchd[13066]: dhcp-service (PID 14763) terminated by signal number 9! Oct 28 08:14:47 Switch jlaunchd[13066]: Registered PID 1699(dhcp-service): exec_command Oct 28 08:14:47 Switch jlaunchd[13066]: dhcp-service (PID 1699) started Oct 28 08:14:47 Switch jlaunchd[13066]: Registered PID 1699(dhcp-service): new process Oct 28 08:14:47 Switch mgd[92629]: UI_CHILD_EXITED: Child exited: PID 1459, status 1, command '/sbin/ifinfo' Oct 28 08:14:50 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_MINOR_EVENT: System under low memory condition Oct 28 08:14:53 Switch jdhcpd: LIBSDB_CLIENT_COUNT_SUPPORT_SCALING_TYPE: Scaling type of this platform is 2 Oct 28 08:14:53 Switch jdhcpd: LIBSDB_MODEL_BASED_CLIENT_COUNT_SUPPORTED: model: 169 supported_clients: 48000 Oct 28 08:14:55 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_MINOR_EVENT: System under low memory condition Oct 28 08:15:06 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_MINOR_EVENT: System under low memory condition Oct 28 08:15:06 Switch vccpd[13423]: JTASK_SCHED_SLIP_KEVENT: 13 sec 364650 usec kevent block Oct 28 08:15:06 Switch mplsoamd[14721]: JTASK_SCHED_SLIP_KEVENT: 13 sec 606344 usec kevent block Oct 28 08:15:46 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_MINOR_EVENT: System under low memory condition Oct 28 08:15:51 Switch jsysmond[14770]: JSYSMOND_LOW_MEM_RECOVERED: System recovered from low memory condition Oct 28 08:16:00 Switch mplsoamd[14721]: JTASK_SCHED_SLIP_KEVENT: 10 sec 694147 usec kevent block Oct 28 08:16:00 Switch vccpd[13423]: JTASK_SCHED_SLIP_KEVENT: 10 sec 178061 usec kevent block Oct 28 08:16:00 Switch dot1xd[14773]: JTASK_SCHED_SLIP_KEVENT: 13 sec 999789 usec kevent block Oct 28 08:16:00 Switch dumpd[1886]: Core and context for jsd saved in /var/tmp/jsd.core-tarball.0.tgz >>>>>>>>>>>>>>>>>>>>>>>>>>> Oct 28 08:16:01 Switch mgd[1491]: UI_CHILD_SIGNALED: Child received signal: PID 1888, signal Terminated, command='/usr/sbin/kmd' Oct 28 08:16:11 Switch jdhcpd: DH_SVC_SENDMSG_FAILURE: sendmsg() from 0.0.0.0 to port 67 at 255.255.255.255 via interface 7 and routing instance default failed: Network is down Oct 28 08:16:37 Switch na-grpcd[95431]: NA_GRPCD_CONFIG_EDIT_FAILURE: Ephemeral DB Edit config: error_code=4, error_message='Deadline Exceeded' Oct 28 08:17:15 Switch jdhcpd: DH_SVC_SENDMSG_FAILURE: sendmsg() from 0.0.0.0 to port 67 at 255.255.255.255 via interface 7 and routing instance default failed: Network is down Oct 28 08:17:16 Switch mplsoamd[14721]: JTASK_SCHED_SLIP_KEVENT: 19 sec 347241 usec kevent block
Reboot will temporarily fix the problem; however the issue might come back.
Deactivate na-grpc : set system processes na-grpc-server disable
Note: When a switch managed by Mist , telemetry collection is disabled at the switch level; however, configuration and control remain fully orchestrated through Mist.
The jsd core dump is fixed in the following versions:
evo:22.3X80-D49-EVO evo:24.2R2-S1-EVO evo:24.4R2-EVO evo:25.2R1-EVO evo:25.3R1-EVO junos:23.4R2-S6 junos:24.2R2-S1 junos:24.4R2 junos:24.4R2-S1 junos:24.4R2-S2 junos:25.2R1 junos:25.3R1