System reaching processes ceiling low watermark: Contact to system administrator to clean up unnecessary processes or increase maxproc ceiling.
It is seen that the following log messages are being reported:
From the RSI we can see that there are 420 zombies created.
jnpr> show system processes extensive no-forwarding
last pid: 48067; load averages: 0.40, 0.23, 0.18 up 406+00:02:30 15:28:09
999 threads: 13 running, 505 sleeping, 420 zombie, 61 waiting
Inorder to find the zombie/defunct process you can do the following.
Start shell user root
# ps axf | grep defu *** this will provide a list of all the zombie PID
1471 - Z 0:00.04 <defunct>
1472 - Z 0:00.00 <defunct>
1901 - Z 0:00.00 <defunct>
1958 - Z 0:00.05 <defunct>
1987 - Z 0:00.04 <defunct>
2983 - Z 0:00.00 <defunct>
2984 - Z 0:00.00 <defunct>
2985 - Z 0:00.00 <defunct>
2986 - Z 0:00.00 <defunct>
To see what the parent PID ( process) is it tied too.
procstat <pid> - pid is from output above
# procstat 2986
PID PPID PGID SID TSID THR LOGIN WCHAN EMUL COMM
2986 28009 21335 21335 0 1 - - FreeBSD ELF32 jdhcpd
# procstat 2987
2987 28009 21335 21335 0 1 - - FreeBSD ELF32 jdhcpd
# procstat 2988
2988 28009 21335 21335 0 1 - - FreeBSD ELF32 jdhcpd
# procstat 2989
2989 28009 21335 21335 0 1 - - FreeBSD ELF32 jdhcpd
# procstat 2990
2990 28009 21335 21335 0 1 - - FreeBSD ELF32 jdhcpd
From CLI you can confirm jdhcpd pid:
> show system processes extensive no-forwarding
28009 root 20 0 629M 112M select 3 865:38 0.00% jdhcpd{jdhcpd}
DHCPD is one of the daemons which forks a child process for the UI commands, and once the child process exits, there is no specific handler to clean it up, either the parent process takes that responsibility, or the system does in case parent ignores the SIGCHLD signal. We want our child process to run in parallel with the parent without it waiting or suspension of execution, in this case we need to ignore the SIGCHLD and that not being implemented for this platform is the root cause of zombie processes lurking around. This issue can be seen when performing the following commands.
Show dhcp relay bindings
Work around – You can restart jdhcpd – restart dhcp-service
https://prsearch.juniper.net/PR1730750