[Junos] Error during commit: "error: configuration database locked by"

Description

Users may find that after executing configuration changes, the commit command fails with the error message " error: configuration database locked by: " or that they are unable to get into Configure mode due to multiple unsuccessful Command Line Interface (CLI) sessions.

This article gives the reason for these problems to occur and details the steps to resolve them.

Symptoms

After configuration changes, the commit command fails with following output:

error: configuration database locked by:
test terminal p2 (pid 22210) on since 2010-12-02 16:14:54 IST, idle 02:23:06
exclusive [edit]

Solution

From the error message, we discover that a user with pid 22210 has been configured in Exclusive mode due to which configuration on the router is not allowed.

Further, no such user appears in the output of the show system users command as shown below:

[edit]
test@test# run show system users 6:40PM up 579 days, 5:01, 1 user, load averages: 0.00, 0.00, 0.00
USER TTY FROM LOGIN@ IDLE WHAT
     test p0 202.163.59.246 6:38PM - -cli (cli)

Nor does the pid 22210 appear in the output of the show process extensive command.

For more information, see show system users .

To resolve this issue, perform the following steps:

Log in to operational prompt and execute the commit command.
Log out user pid 22210 with the following command from the operational prompt:

request system logout pid <pid>

Example:

srx> request system logout pid 22210

For more information about this command, see request system logout .

If provisioning tools are used to connect remotely, lock the database, and then commit the changes. It is possible that the TCP socket disconnects abruptly due to network issues. And the subsequent unsuccessful attempts by the provisioning tool will cause too many mgd processes to spawn and result in Configure mode not being accessible at all.

The following session log explains how to clear all those mgd sessions to know the failed PID that locked the DB, and then get access to the CLI Configure mode.

Example (from Primary):

jtac@jtac_lab> show system processes wide detail |match "PID|mgd"
  PID  UID  PPID CPU  PRI  NI     RSS WCHAN    STARTED TT  STAT         TIME COMMAND
  247    0  3389   0   20   0   22612 lockf     7:56AM  -  Is        0:00.01 mgd: (mgd) (remote_provision)/dev/pts/0 (mgd)
  252    0     1   0   21   0   25420 lockf     7:57AM  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
  267    0     1   0   21   0   25420 lockf     8:07AM  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
  282    0     1   0   21   0   25420 lockf     8:16AM  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
  301    0     1   0   20   0   26480 lockf     8:28AM  -  Is        0:25.51 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
 3389    0  3372   0   20   0   21648 select   28Dec18  -  S         0:49.09 /usr/sbin/mgd -N
 3417    0  3372   0   20   0    5564 select   28Dec18  -  I         0:00.02 /usr/sbin/appidd -N
 3435    0  3372   0   20   0    8340 select   28Dec18  -  I         0:00.23 /usr/sbin/bbe-smgd -b -N
 5645    0  3389   0   20   0   22324 lockf     8:58AM  -  Is        0:00.01 mgd: (mgd) (whartnett)/dev/pts/0 (mgd)
33166    0     1   0   21   0   25420 lockf     1:54PM  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
33399    0     1   0   21   0   25420 lockf     2:03PM  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
33419    0     1   0   21   0   25420 lockf     2:12PM  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
33440    0     1   0   21   0   25420 lockf     2:21PM  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
33453    0     1   0   21   0   25420 lockf     2:30PM  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
33469    0     1   0   21   0   25424 lockf     2:45PM  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
33480    0     1   0   21   0   25424 lockf     2:58PM  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
33716    0     1   0   21   0   25424 lockf     3:13PM  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
33737    0     1   0   52   0   25964 lockf     3:23PM  -  Is        0:05.71 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
34035    0 34032   0   21   0   25424 lockf     3:38PM  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
34053    0  3389   0   20   0   22504 lockf     3:47PM  -  Is        0:00.01 mgd: (mgd) (jtac)/dev/pts/0 (mgd)
34058    0  3389   0   20   0   23772 select    3:48PM  -  Ss        0:00.26 mgd: (mgd) (jtac)/dev/pts/1 (mgd)
34083    0 34058   0   20   0    2356 -         3:50PM  -  R         0:00.01 /bin/ps -ax -rOuid,ppid,cpu,pri,nice,rss,wchan,start -ww
91211    0     1   0   20   0   27616 select    6:03AM  -  Ss        1:25.77 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
99333    0     1   0   40   0   25364 lockf     7:15AM  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
99352    0     1   0   52   0   26028 lockf     7:24AM  -  Is        0:05.64 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
99635    0     1   0   52   0   25856 lockf     7:32AM  -  Is        0:05.22 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
99896    0     1   0   21   0   25536 lockf     7:40AM  -  Is        0:00.15 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
99912    0     1   0   21   0   25420 lockf     7:48AM  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)

jtac@jtac_lab>

jtac@jtac_lab> show system processes extensive |match mgd |match lock
  301 root             1  20    0   724M 26480K lockf   2   0:26   0.00% mgd
33737 root             1  52    0   724M 25964K lockf   3   0:06   0.00% mgd
99352 root             1  52    0   724M 26028K lockf   1   0:06   0.00% mgd
99635 root             1  52    0   724M 25856K lockf   3   0:05   0.00% mgd
99896 root             1  21    0   724M 25536K lockf   3   0:00   0.00% mgd
33716 root             1  21    0   724M 25424K lockf   3   0:00   0.00% mgd
  267 root             1  21    0   724M 25420K lockf   0   0:00   0.00% mgd
99912 root             1  21    0   724M 25420K lockf   1   0:00   0.00% mgd
34035 root             1  21    0   724M 25424K lockf   2   0:00   0.00% mgd
33453 root             1  21    0   724M 25420K lockf   2   0:00   0.00% mgd
  252 root             1  21    0   724M 25420K lockf   0   0:00   0.00% mgd
33469 root             1  21    0   724M 25424K lockf   2   0:00   0.00% mgd
  282 root             1  21    0   724M 25420K lockf   1   0:00   0.00% mgd
33480 root             1  21    0   724M 25424K lockf   1   0:00   0.00% mgd
33440 root             1  21    0   724M 25420K lockf   2   0:00   0.00% mgd
33419 root             1  21    0   724M 25420K lockf   2   0:00   0.00% mgd
33399 root             1  21    0   724M 25420K lockf   1   0:00   0.00% mgd
33166 root             1  21    0   724M 25420K lockf   3   0:00   0.00% mgd
99333 root             1  40    0   724M 25364K lockf   0   0:00   0.00% mgd
34053 root             1  20    0   724M 22504K lockf   1   0:00   0.00% mgd
  247 root             1  20    0   724M 22612K lockf   2   0:00   0.00% mgd
5645 root             1  20    0   724M 22324K lockf   1   0:00   0.00% mgd

root@jtac_lab:/var/home/superUserClass # kill 247
root@jtac_lab:/var/home/superUserClass # ps ax | grep mgd

  252  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
  267  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
  282  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
  301  -  Is        0:25.51 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
 3389  -  I         0:49.11 /usr/sbin/mgd -N
 3435  -  I         0:00.23 /usr/sbin/bbe-smgd -b -N
 5645  -  Is        0:00.01 mgd: (mgd) (whartnett)/dev/pts/0 (mgd)
33166  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
33399  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
33419  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
33440  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
33453  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
33469  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
33480  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
33716  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
33737  -  Is        0:05.71 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
34035  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
34053  -  Is        0:00.01 mgd: (mgd) (jtac)/dev/pts/0 (mgd)
34058  -  Is        0:00.32 mgd: (mgd) (jtac)/dev/pts/1 (mgd)
34125  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
34154  -  Is        0:00.29 mgd: (mgd) (jtac)/dev/pts/2 (mgd)
34173  -  Is        0:00.01 mgd: (mgd) (jtac)/dev/pts/2 (mgd)
34177  -  Ss        0:00.18 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
34300  -  Is        0:00.01 mgd: (mgd) (jtac)/dev/pts/3 (mgd)
91211  -  Ss        1:25.78 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
99333  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
99352  -  Is        0:05.64 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
99635  -  Is        0:05.22 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
99896  -  Is        0:00.15 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
99912  -  Is        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
34434  3  S+        0:00.00 grep mgd

root@jtac_lab:/var/home/superUserClass # kill 252
root@jtac_lab:/var/home/superUserClass # kill 267
root@jtac_lab:/var/home/superUserClass # kill 282
root@jtac_lab:/var/home/superUserClass # kill 301
root@jtac_lab:/var/home/superUserClass # kill 5645
root@jtac_lab:/var/home/superUserClass # kill 33166
root@jtac_lab:/var/home/superUserClass # kill 33399
root@jtac_lab:/var/home/superUserClass # kill 33419
root@jtac_lab:/var/home/superUserClass # kill 33440
root@jtac_lab:/var/home/superUserClass # kill 33453
root@jtac_lab:/var/home/superUserClass # kill 33469
root@jtac_lab:/var/home/superUserClass # kill 33480
root@jtac_lab:/var/home/superUserClass # kill 33716
root@jtac_lab:/var/home/superUserClass # kill 33737
root@jtac_lab:/var/home/superUserClass # kill 34035
root@jtac_lab:/var/home/superUserClass # kill 34125
root@jtac_lab:/var/home/superUserClass # kill 34177
root@jtac_lab:/var/home/superUserClass # kill 91211
root@jtac_lab:/var/home/superUserClass # ps ax | grep mgd

 3389  -  I         0:49.11 /usr/sbin/mgd -N
 3435  -  I         0:00.23 /usr/sbin/bbe-smgd -b -N
34053  -  Is        0:00.12 mgd: (mgd) (jtac)/dev/pts/0 (mgd)
34058  -  Is        0:00.32 mgd: (mgd) (jtac)/dev/pts/1 (mgd)
34154  -  Is        0:00.29 mgd: (mgd) (jtac)/dev/pts/2 (mgd)
34173  -  Is        0:00.01 mgd: (mgd) (jtac)/dev/pts/2 (mgd)
34300  -  Is        0:00.01 mgd: (mgd) (jtac)/dev/pts/3 (mgd)
34447  3  S+        0:00.00 grep mgd

root@jtac_lab:/var/home/superUserClass # cli

jtac@jtac_lab> show system processes extensive |match mgd
 3389 root             1  20    0   724M 21648K select  1   0:49   0.00% mgd
34058 root             1  20    0   724M 23908K select  2   0:00   0.00% mgd
34154 root             1  20    0   724M 23872K select  0   0:00   0.00% mgd
 3435 root             1  20    0   765M  8340K select  2   0:00   0.00% bbe-smgd
34053 root             1  21    0   724M 22676K select  3   0:00   0.00% mgd
34300 root             1  20    0   724M 22380K select  3   0:00   0.00% mgd
34173 root             1  20    0   724M 22432K select  1   0:00   0.00% mgd
34449 root             1  20    0   724M 22596K select  1   0:00   0.00% mgd

jtac@jtac_lab> exit

root@jtac_lab:/var/home/superUserClass # kill 34058
root@jtac_lab:/var/home/superUserClass # kill 34154
root@jtac_lab:/var/home/superUserClass # kill 34053
root@jtac_lab:/var/home/superUserClass # kill 34300
root@jtac_lab:/var/home/superUserClass # kill 34173
root@jtac_lab:/var/home/superUserClass #
root@jtac_lab:/var/home/superUserClass #
root@jtac_lab:/var/home/superUserClass # ps ax | grep mgd

 3389  -  I         0:49.11 /usr/sbin/mgd -N
 3435  -  I         0:00.23 /usr/sbin/bbe-smgd -b -N
34454  -  Ss        0:00.14 mgd: (mgd) /packages/mnt/junos-runtime32-563987c0/usr/sbin/mgd (remote_provision) (mgd)
34458  3  S+        0:00.00 grep mgd

root@jtac_lab:/var/home/superUserClass # kill 34454
root@jtac_lab:/var/home/superUserClass #
root@jtac_lab:/var/home/superUserClass # ps ax | grep mgd

 3389  -  S         0:49.11 /usr/sbin/mgd -N
 3435  -  I         0:00.23 /usr/sbin/bbe-smgd -b -N
34460  3  S+        0:00.00 grep mgd

root@jtac_lab:/var/home/superUserClass #
root@jtac_lab:/var/home/superUserClass #

At this point, users can get into Configuration mode:

root@jtac_lab:/var/home/superUserClass # cli

jtac@jtac_lab> configure
Entering configuration mode
Users currently editing the configuration:
  remote_provision (pid 91211) on since 2019-06-03 07:06:52 EDT, idle 09:09:50  >>> The problematic session with a lock that was abruptly disconnected
      exclusive
  jtac terminal pts/0 (pid 34053) on since 2019-06-03 16:14:44 EDT, idle 00:01:58
      private [edit]
The configuration has been changed but not committed

[edit]
jtac@jtac_lab# quit
The configuration has been changed but not committed
Exiting configuration mode

jtac@jtac_lab> request system logout pid 91211
warning: discarding uncommitted changes

jtac@jtac_lab>

jtac@jtac_lab>

[edit]
jtac@jtac_lab# exit
Exiting configuration mode

jtac@jtac_lab> show system processes extensive |match mgd
34492 root             1  21    0   724M 23848K select  0   0:00   1.46% mgd
 3389 root             1  20    0   724M 21648K select  1   0:49   0.00% mgd
 3435 root             1  20    0   765M  8340K select  2   0:00   0.00% bbe-smgd
34467 root             1  20    0   724M 25536K select  1   0:00   0.00% mgd

jtac@jtac_lab>

Modification History

2023-11-15: Article reviewed for accuracy; it is valid and accurate

Related Information

KB27969 - [SRX] Commit fails in SRX chassis cluster with dual REs on each node [juniper.net]