Description

mgd and phone-home crashing on EX3400s after tls ca-profiles configurations were added

Symptoms

MGD and phone-home core files seen on EX3400 device:

{master:0}
user@hostname> show system core-dumps 
fpc0:
--------------------------------------------------------------------------
/var/crash/*core*: No such file or directory
-rw-rw----  1 root  wheel     764611 Mar 17 21:01 /var/tmp/eventd.core-tarball.0.tgz <<< KB78081 created for these eventd core files
-rw-rw----  1 root  wheel     708128 Mar 18 13:46 /var/tmp/eventd.core-tarball.1.tgz
-rw-------  1 root  wheel    6352422 Mar 18 13:46 /var/tmp/mgd.core.0.gz <<<<<<<<<<<<<<
-rw-------  1 root  wheel    6322483 Mar 18 13:47 /var/tmp/mgd.core.1.gz <<<<<<<<<<<<<<
-rw-------  1 root  wheel     223858 Mar 18 09:15 /var/tmp/phone-home.core.0.gz <<<<<<<<<
/var/tmp/pics/*core*: No such file or directory
/var/crash/kernel.*: No such file or directory
/var/jails/rest-api/tmp/*core*: No such file or directory
/tftpboot/corefiles/*core*: No such file or directory
total files: 5

The root cause seems to be phone-homed packages are getting deleted. 
Now when mgd is exiting it calls exit to do some cleanup and since the packages are not present exit is coring in attempt to clean shared libraries related with the packages.

Looks like this packages are getting deleted abrubtly as symlinks are still present:

user@hostname:/usr/lib/dd # ls -ltr /usr/lib/render/libphone-home-render.tlv
lrwxr-xr-x 1 root wheel 65 Dec 27 05:40 /usr/lib/render/libphone-home-render.tlv -> /packages/mnt/jphone-home/usr/lib/render/libphone-home-render.tlv

user@hostname:/usr/lib/dd # ls -ltr /packages/mnt/jphone-home/usr/lib/render/libphone-home-render.tlv
ls: /packages/mnt/jphone-home/usr/lib/render/libphone-home-render.tlv: No such file or directory


What triggered deletion of this packages is not known(at least from the logs available). 
  • As from mgd point of view we didn't see anyone triggering this deletion.
  • Moreover, mgd on itself does not have authorisation to add/delete any package.

 

Solution

One preventive fix is committed with PR 1707355 in 21.4R3S7.
This issue is being investigated through PR1795742 which has been marked as a duplicate of PR1707355.

After upgrading to 21.4R3S7 the mgd and phone-home are not seen. 

Modification History

2024-03-20 : Article Created