Description

When Junos OS is upgraded on backup RE with GRES disabled, the validation fails and the following error message appears:

Validating against /config/juniper.conf.gz Local Policy Decision Function: LPDFD failed to initialize the shared-memory.
mgd: error: configuration check-out failed

This article explains the details of the issue and provides a solution.

Symptoms

To reproduce the problem, follow these steps:

  1. Deactivate chassis redundancy graceful-switchover on primary RE for upgrade preparation

  2. Note that the following files have been deleted manually before this upgrade on backup RE:

    /var/tmp/shm_ipc_sem_<*>

  3. Log in to the backup RE and execute the command "request system software add  jinstall- <version> "

  4. The following error message is reported and validation fails:
Validating against /config/juniper.conf.gz Local Policy Decision Function: LPDFD failed to initialize the shared-memory.
mgd: error: configuration check-out failed

Solution

The cause is related to the leak of system v kernel semaphores and default value is 10. The shared-memory system call returns errors when the limit is exceeded.

In Junos OS 9.3, 6 out of 10 semaphores are part of normal allocations every time the host is booted. In /var/tmp , shm-ipc semaphore files named shm_ipc_sem* represent those semaphores. If somehow those files get deleted and one of the related daemons (dcd, cosd, dfwd, rpd) is restarted, (for example, disabled GRES on primary) the routing engine (RE) will cause these daemons restart on the backup RE. This action will introduce a leak of 1 semaphore. These files do not get deleted on a request system storage cleanup . The only way to delete these files is if someone executes a rm command from the /var/tmp directory.

One way to find out if these files got deleted for the life time of the router is to look at the files' timestamps. The timestamp should match the router boot up time.

root@router% ls -l /var/tmp/shm_ipc_sem_*
-rw-r--r-- 1 root field 0 Jun 5 15:54 /var/tmp/shm_ipc_sem_1   <<< Jun 5 15:54 around should be the router boot up time, if not it means these files get deleted.
-rw-r--r-- 1 root field 0 Jun 5 15:54 /var/tmp/shm_ipc_sem_2
-rw-r--r-- 1 root field 0 Jun 5 15:54 /var/tmp/shm_ipc_sem_3
-rw-r--r-- 1 root field 0 Jun 5 15:54 /var/tmp/shm_ipc_sem_4

A graceful routing engine switchover (GRES) switchover does not recover this semaphore leak. The only way to recover is to reboot the routing engine.

As a workaround, just perform a GRES and reboot the new backup. Then the validation of the Junos OS upgrade should pass.