Description

This article explains the issue when multiple commits are performed, the following repetitive log message is seen: 

'commit confirmed will be rolled back in less than 1 minute'

Symptoms

User cannot make any configuration changes on the device and sessions are hung after entering into configuration mode: 

{MASTER}
user@MX> 
# commit confirmed will be rolled back in less than 1 minute
{MASTER}
user@MX> 
# commit confirmed will be rolled back in less than 1 minute
{MASTER}
user@MX>

Solution

This issue occurs due to the commit process being stuck, which leads to the MGD getting stuck (MGD takes care of the commits).

user@MX> show system processes extensive | match mgd
18311 root       20    0   731M 30308K select  1   1:05   0.00% mgd
45552 root       40    0   758M 45436K lockf   2   0:06   0.00% mgd
47000 root       20    0   750M 32740K lockf   1   0:03   0.00% mgd
49980 root       27    0   758M 42016K select  1   0:02   0.00% mgd
49616 root       20    0   750M 32672K select  2   0:01   0.00% mgd
18398 root       20    0   874M  8844K select  1   0:00   0.00% bbe-smgd
46974 root       20    0   750M 31264K lockf   1   0:00   0.00% mgd
49837 root       20    0   750M 31300K lockf   3   0:00   0.00% mgd
46651 root       20    0   731M 30752K lockf   2   0:00   0.00% mgd
19125 root       25    0   717M  5520K select  3   0:00   0.00% mgd-api
50182 root       20    0   750M 31192K select  2   0:00   0.00% mgd

This can happen when there is some issue with the /var/ directory of the RE SSD, as any command that invokes the MGD process makes the MGD process look into the schema.db file. If there is any issue with the /var/ directory, the schema file could not be fetched. Hence, the process gets stuck.

Perform the steps below  during a maintenance window to resolve the issue

  1. Access the shell and  % mgd -I to restart the management process. Then check whether this fixes the issue.

  2. Perform a mastership switchover

  3. Once RE 0 comes back up, reboot the RE0

  4. Switchover the mastership back to RE0

{MASTER}
user@MX> start shell user root
Password:
root@MX:/var/home/remote # mgd -I

# commit confirmed will be rolled back in less than 1 minute
{MASTER}
user@MX> request chassis routing-engine master switch
Toggle mastership between routing engines ? [yes,no] (no) yes

Resolving mastership...

{MASTER}
user@MX> show chassis routing-engine
Routing Engine status:
 Slot 0:
    Current state                  Backup
    Election priority              Master (default)
    
Routing Engine status:
  Slot 1:
    Current state                  Master
    Election priority              Backup (default)

{MASTER}
user@MX> request routing-engine login other-routing-engine

--- JUNOS 17.3R3.10 Kernel 64-bit  JNPR-10.3-20180615.4674357_buil
# commit confirmed will be rolled back in less than 1 minute
{BACKUP.EN_US}

{BACKUP.EN_US}
user@MX> request system reboot
Reboot the system ? [yes,no] (no) yes

*** FINAL System shutdown message from remote@MX ***

System going down IMMEDIATELY

{MASTER}
root@MX> edit
Entering configuration mode
{MASTER}
user@MX# commit 
re1: 
configuration check succeeds
re0: 
commit complete
re1: 
commit complete