Description

Executing "commit confirmed" needs to have a "commit sync" within 10 minutes to actually commit the new configuration to the two REs. Otherwise, the new configuration will be rolled back by the system. However, there's a situation where the "commit sync" was executed on the primary RE in the last seconds/miliseconds whereas the backup RE was rolling back the configuration at the same time. In this situation, the "commit sync" will fail on the backup RE and following was seen: 

# commit confirmed will be rolled back in 10 minutes
[MASTER][edit]
labroot@mx960-re0# commit synchronize
Jul 14 20:12:32
re0:
re1: 
error: configuration database locked by:  initial terminal  (pid 14809) on since 2020-07-14 20:12:32 CST      exclusive 
re0: 
error: remote lock-configuration failed on re1

 

Symptoms

Test Example: 
[MASTER][edit]
mx960-re0# set logical-systems test9   <<< new configuration added
Jul 15 09:29:33

[MASTER][edit]
mx960-re0# commit confirmed  <<< bring the new configuration to both REs
Jul 15 09:29:35
re0: 
configuration check succeeds
commit confirmed will be automatically rolled back in 10 minutes unless confirmed
re1: 
commit complete
commit confirmed will be automatically rolled back in 10 minutes unless confirmed
re0: 
commit complete

# commit confirmed will be rolled back in 10 minutes
[MASTER][edit]
mx960-re0# show | compare rollback 0 
Jul 15 09:30:00

# commit confirmed will be rolled back in 10 minutes
[MASTER][edit]
mx960-re0# show | compare rollback 1    
Jul 15 09:30:04
[edit logical-systems]
+  test9;     <<< the new configuration saved into "rollback 1" on RE0

# commit confirmed will be rolled back in 9 minutes
[BACKUP][edit]
mx960-re1# show | compare rollback 0 

# commit confirmed will be rolled back in 9 minutes
[BACKUP][edit]
mx960-re1# show | compare rollback 1    
[edit logical-systems]
+  test9; <<< the new configuration saved into "rollback 1" on RE1

mx960-re0# commit synchronize 
Jul 15 09:39:41
re0: 
re1: 
error: configuration database locked by:  initial terminal  (pid 20534) on since 2020-07-15 09:39:41 CST      exclusive 
re0: 
error: remote lock-configuration failed on re1
note: consider using 'commit synchronize force' to
terminate remote edit sessions and force the commit
<<< error occured

On RE0:
[MASTER][edit]
mx960-re0# show | compare rollback 0 
Jul 15 09:39:59

[MASTER][edit]
mx960-re0# show | compare rollback 1    
Jul 15 09:40:03
[edit logical-systems]
+  test9;

mx960-re0# show logical-systems 
Jul 15 09:41:38
test9;  <<< configuration already here
The new configuration commit succeeded and saved into rollback 1

On RE1:
Broadcast Message from root@tm-static-mx960-r2028-re1                          
        (no tty) at 9:39 CST...  
Commit was not confirmed; automatic rollback complete.   <<< configuration rollbacked                                                                            
[BACKUP][edit]
mx960-re1# show | compare rollback 0 

[BACKUP][edit]
mx960-re1# show | compare rollback 1    
[edit logical-systems]
-  test9;

[BACKUP][edit]
mx960-re1# show logical-systems 

[BACKUP][edit]
The current configuration does not include the new configuration because it has been rolled back.

Then I did a commit syn on RE0 couple minutes later and succeeded:
mx960-re0# commit synchronize   <<< 
Jul 15 09:42:00
re0: 
configuration check succeeds
re1: 
commit complete
re0: 
commit complete

[MASTER][edit]
labroot@-mx960-re0# run show system commit   
Jul 15 09:42:12
0   2020-07-15 09:42:04 CST by labroot via cli commit synchronize
1   2020-07-15 09:29:40 CST by labroot via cli commit confirmed, rollback in 10mins synchronize
 
[BACKUP][edit]
labroot@mx960-re1# run show system commit   
0   2020-07-15 09:42:03 CST by labroot via synchronize
1   2020-07-15 09:39:42 CST by root via other <<< rollback succeeded at this time
2   2020-07-15 09:29:39 CST by labroot via synchronize commit confirmed, rollback in 10mins

The first "commit sync" executed on RE0 was on "09:39:41" and failed due to RE1 being locked. The rollback of RE1 only finished on 09:39:42. That's why the first commit sync failed.

Solution

This is due to the "commit sync" executed during the last seconds of the 10 minutes time of rollback period. Another "commit sync" will be succeeded a few minutes later and synchronized configuration of both REs.

Also in this situation, the configuration in the primary and backup RE are different. Primary RE has new configuration and the backup RE has the rolled back configuration. Therefore, another "commit sync" a few minutes later without seeing any error is needed to synchronize the configuration on both REs.

A rollback earlier than that is highly recommended to avoid this issue.

Modification History

2021-03-25: Updated the article terminology to align with Juniper's Inclusion & Diversity initiatives