This article explains a Junos software bug where after ephemeral commit changes, multiple daemons start crashing and they keep crashing until a possible mitigation is applied.
Multiple daemons crash seen, after committing Ephemeral DB config changes. The core dump reason for these crashes would be 'segmentation fault'.
show system core-dumps no-forwarding
/var/crash/*core*: No such file or directory
-rw------- 1 root wheel 7020502 Jun 5 17:06 /var/tmp/alarm-mgmtd.core.0.gz
-rw------- 1 root wheel 7020653 Jun 5 17:15 /var/tmp/alarm-mgmtd.core.1.gz
-rw------- 1 root wheel 6990042 Jun 5 17:06 /var/tmp/alarmd.core.0.gz
-rw------- 1 root wheel 6990199 Jun 5 17:15 /var/tmp/alarmd.core.1.gz
-rw------- 1 root wheel 6606893 Jun 5 17:06 /var/tmp/aprobed.core.0.gz
-rw------- 1 root wheel 6607056 Jun 5 17:15 /var/tmp/aprobed.core.1.gz
-rw------- 1 root wheel 6628112 Jun 5 17:09 /var/tmp/auditd.core.0.gz
-rw------- 1 root wheel 6628194 Jun 5 17:18 /var/tmp/auditd.core.1.gz
-rw------- 1 root wheel 7418942 Jun 5 17:02 /var/tmp/bbe-smgd.core.0.gz
-rw------- 1 root wheel 7418856 Jun 5 17:08 /var/tmp/bbe-smgd.core.1.gz
-rw------- 1 root wheel 7417708 Jun 5 17:16 /var/tmp/bbe-smgd.core.2.gz
-rw------- 1 root wheel 7418075 Jun 5 16:48 /var/tmp/bbe-smgd.core.3.gz
-rw------- 1 root wheel 7418374 Jun 5 16:55 /var/tmp/bbe-smgd.core.4.gz
-rw------- 1 root wheel 7011413 Jun 5 17:09 /var/tmp/bbe-statsd.core.0.gz
-rw------- 1 root wheel 7012604 Jun 5 17:18 /var/tmp/bbe-statsd.core.1.gz
-rw------- 1 root wheel 6628362 Jun 5 17:08 /var/tmp/bdbrepd.core.0.gz
Jun 5 16:30:32.255 <router-name> mgd[78095]: %INTERACT-4-UI_EPHEMERAL_COMMIT_COMPLETED: commit complete on 'default' ephemeral database
Jun 5 16:39:37.227 <router-name> mgd[9367]: %DAEMON-3-UI_CHILD_SIGNALED: Child received signal: PID 37240, signal Segmentation fault, core dumped, command='/sbin/ifinfo'
Jun 5 16:39:42.425 <router-name> mgd[37252]: %DAEMON-3-UI_CHILD_SIGNALED: Child received signal: PID 37287, signal Segmentation fault, core dumped, command='/sbin/ifinfo'
Jun 5 16:39:49.686 <router-name> mgd[37296]: %DAEMON-3-UI_CHILD_SIGNALED: Child received signal: PID 37298, signal Segmentation fault, core dumped, command='/sbin/ifinfo'
Jun 5 16:40:05.194 <router-name> mgd[37307]: %DAEMON-3-UI_CHILD_SIGNALED: Child received signal: PID 37334, signal Segmentation fault, core dumped, command='/usr/sbin/rpd'
Jun 5 16:40:05.229 <router-name> mgd[37307]: %DAEMON-3-UI_CHILD_SIGNALED: Child received signal: PID 37335, signal Segmentation fault, core dumped, command='/usr/sbin/chassisd'
Jun 5 16:40:21.675 <router-name> mgd[37307]: %DAEMON-3-UI_CHILD_SIGNALED: Child received signal: PID 37363, signal Segmentation fault, core dumped, command='/usr/sbin/snmpd'
This is a software bug fixed via - PR 1539455. This happens after any commit is performed for the Ephemeral database.
Root cause – While adding certain checks to avoid setting the change bits to configuration objects in daemon context in the presence of ephemeral database, a string comparison was done without validating a variable for NULL value, thus leading to a core when variable has NULL value. This was done assuming variable will be set appropriately before being used for comparison, which was not the case and hence the issue.
Fix – Appropriate NULL check was added to prevent the core for the case where variable will not be set before performing the string comparisons.
There is no mitigation/workaround available to avoid this problem. If Ephemeral DB changes are avoided, the problem can be avoided.
This is not a day-1 issue and was introduced as regression of PR1520936.
----------------------
Replication Steps
labroot@atlas-re0> edit ephemeral
Jun 07 11:09:05
warning: uncommitted changes will be discarded on exit
Entering configuration mode
{master}[edit]
labroot@atlas-re0# commit
Jun 07 11:09:07
commit complete
labroot@atlas-re0> show log syslog | match dumped | last 20 | except labroot
Jun 07 11:10:47
Jun 7 11:09:26.402 atlas-re0 mgd[24133]: %DAEMON-3-UI_CHILD_SIGNALED: Child received signal: PID 84080, signal Segmentation fault, core dumped, command='/sbin/ifinfo'
Jun 7 11:09:41.471 atlas-re0 mgd[24133]: %DAEMON-3-UI_CHILD_SIGNALED: Child received signal: PID 84089, signal Segmentation fault, core dumped, command='/sbin/ifinfo'
Jun 7 11:09:56.572 atlas-re0 mgd[24133]: %DAEMON-3-UI_CHILD_SIGNALED: Child received signal: PID 84096, signal Segmentation fault, core dumped, command='/sbin/ifinfo'
Jun 7 11:10:13.442 atlas-re0 mgd[24133]: %DAEMON-3-UI_CHILD_SIGNALED: Child received signal: PID 84098, signal Segmentation fault, core dumped, command='/sbin/ifinfo'
Jun 7 11:10:22.819 atlas-re0 mgd[84104]: %DAEMON-3-UI_CHILD_SIGNALED: Child received signal: PID 84120, signal Segmentation fault, core dumped, command='/usr/sbin/rpd'
Jun 7 11:10:23.459 atlas-re0 mgd[84104]: %DAEMON-3-UI_CHILD_SIGNALED: Child received signal: PID 84121, signal Segmentation fault, core dumped, command='/usr/sbin/chassisd'