Description

This KB is explaining the log messages related to database version number mismatch, when copy-config, get-configuration, and discard-change RPCs run in two parallel NETCONF sessions, below are the logs snippet:
 

Mar 12 05:02:05 lab-device mgd[66312]: %DAEMON-0: ../../../../../src/ui/lib/memory/getmem.c:559: insist '!"mmap: could not remap file"' failed

Mar 12 05:02:16 lab-device mgd[66318]: %DAEMON-3-UI_DBASE_MISMATCH_MAJOR: Database header major version number mismatch for file '/var/run/db/juniper.db': expecting 0x1c, got 0x1e

Mar 12 05:02:16 lab-device mgd[66341]: %DAEMON-3-UI_DBASE_MISMATCH_MAJOR: Database header major version number mismatch for file '/var/run/db/juniper.db': expecting 0x1c, got 0x1e

Symptoms

On all Junos OS and Junos OS Evolved devices, when copy-config, get-configuration, and discard-change RPCs run in two parallel NETCONF sessions and the database is also accessed in parallel by two NETCONF sessions, it may lead to database corruption and mgd-related services crash.  Below log message and core-dump will be seen during issue state:
 

Coredump: 
-rw-rw---- 1 root wheel  5634138 Mar 12 05:02 /var/tmp/mgd.core.4.gz 


Below log snippets are from QFX-5100 device, which shows 2 different Netconf sessions and it lead to DB corruption: 

Mar 12 05:02:03 lab-device mgd[66313]: %INTERACT-6-UI_AUTH_EVENT: Authenticated user 'User-A' assigned to class 'j-admin-access'

Mar 12 05:02:03 lab-device mgd[66313]: %INTERACT-6-UI_LOGIN_EVENT: User 'User-A' login, class 'j-admin-access' [66313], ssh-connection '100.xx.xx.1xx 43xxx 100.xx.xx.xx 22', client-mode 'cli'
Mar 12 05:02:03 lab-device mgd[66313]: %INTERACT-6-UI_CMDLINE_READ_LINE: User 'User-A', command 'xml-mode netconf need-trailer '
Mar 12 05:02:03 lab-device mgd[66313]: %INTERACT-6-UI_LOGOUT_EVENT: User 'User-A' logout
Mar 12 05:02:03 lab-device mgd[66312]: %INTERACT-6-UI_AUTH_EVENT: Authenticated user 'User-A' assigned to class 'j-admin-access'
Mar 12 05:02:03 lab-device mgd[66312]: %INTERACT-6-UI_LOGIN_EVENT: User 'User-A' login, class 'j-admin-access' [66312], ssh-connection '100.xx.xx.xx 43xxx 100.xx.xx.xx 22', client-mode 'netconf'
Mar 12 05:02:05 lab-device mgd[66312]: %INTERACT-6-UI_NETCONF_CMD: User 'User-A' used NETCONF client to run command 'get-commit-information'
Mar 12 05:02:05 lab-device mgd[66312]: %INTERACT-6-UI_NETCONF_CMD: User 'User-A' used NETCONF client to run command 'lock cannot reconstruct arguments'


DB Corruption Logs:

Mar 12 05:02:16 lab-device mgd[66318]: %DAEMON-3-UI_DBASE_MISMATCH_MAJOR: Database header major version number mismatch for file '/var/run/db/juniper.db': expecting 0x1c, got 0x1e
Mar 12 05:02:16 lab-device mgd[66341]: %DAEMON-3-UI_DBASE_MISMATCH_MAJOR: Database header major version number mismatch for file '/var/run/db/juniper.db': expecting 0x1c, got 0x1e
 

 

Solution

This event indicates that the config db version has changed probably after some activity and hence the event is logged.  There may be a config change being pushed over netconf channel when the error was hit. This is most likely related to database corruption issue that occurs when database is being accessed over 2 or more netconf channels in parallel. 

Recovery:
 
1. Reboot the device using  below command during a MW. This should re-initialize the database and resolve the database mismatch issue
- request system reboot hypervisor
2. If unable to enter edit exclusive session, execute below command, it will clear stale PID and allow the session.
- request system logout PID

Below public PR explains more details and provide fixed versions information: 

Modification History

2023-08-10: Initial Version

Related Information

  • Once the mgd process is killed that is used for the NETCONF session, the system does not allow the user to enter "configure exclusive" via a new SSH session or a new SSH NETCONF session. This is a timing issue and is normally unlikely to be hit.
  • If unable to enter edit exclusive session, "request system logout PID" will clear stale PID and allow the session.