Description

In certain circumstances, commit synchronize might take a long time to complete on a dual routing engine (RE) JUNOS router. Simple commit of the same configuration on either RE takes much less time.

Symptoms

When this happens, the commit synchronize | display detail command shows that most of the delay comes from the step push configuration to re1 

user@host-re0# commit synchronize | display detail | no-more    
re0: 
2009-01-07 11:09:36 CET: obtaining db lock on  re1
.....
2009-01-07 11:09:46 CET: copying file '//var/etc/stp_context.id' to 're1'
2009-01-07 11:09:46 CET: push configuration to re1
2009-01-07 11:10:49 CET: asking re1 to commit
re1: 
2009-01-07 11:11:07 CET: exporting juniper.conf
.....
commit complete
re0: 
2009-01-07 11:11:09 CET: commit wrapup...

Solution

This happens because, during a commit synchronize on the primary the backup RE is trying to open a connection with the accounting server but it is not getting any answer.  An examination of the log messages on the backup RE shows that the following message is logged when a commit synchronize is entered on the master. 

Jan  7 11:30:50  host-re1 mgd[13458]: UI_TACPLUS_ERROR: TACACS+ failure: connect: timed out

This shows that the backup RE is waiting for the connection to time out. It follows that the delay is bigger if there are more unreachable accounting servers in the configuration, and this is indeed what was observed in the lab. 
user@host-re0# show system accounting 
events [ login change-log interactive-commands ];
destination {
    tacplus {
        server {
            10.0.0.1;
            10.0.0.2;
            10.0.0.3;
        }
    }
}

[edit]
user@host-re0#

The root causes behind these timeouts may be one of the following:
  • The backup RE is attempting to open connections to the accounting servers (through fxp0), but getting no reply. The workaround in this case is to deactivate the accounting.
  • A discard next-hop entry exists in the forwarding table, from the backup RE point of view, causing it to silently drop the packets towards the accounting servers. This can be easily verified with the examination of the forwarding table entries on the backup RE with show route forwarding-table 
    user@host-re1>show route forwarding-table destination 10.0.0.0/24 table default                     
Routing table: inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
10.0.0.0/24        user     3                    dscd    34     5

user@host-re1>

    Deactivation of the accounting does resolve the issue here as well; however, a better workaround is to reload the backup routing engine. After the reload, the discard next-hop entry disappears from the forwarding table. A default entry with a reject next-hop is used in its place. 
    user@host-re1>show route forwarding table destination 10.0.0.0/24 table default             
Routing table: inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            perm     3                    rjct    36     4

user@host-re1>

    When a commit synchronize is performed on the primary RE, the backup RE now logs the following message: 
    Jan  7 13:48:44  host-re1 mgd[4612]: UI_TACPLUS_ERROR: TACACS+ failure: connect: No route to host

    In the log message, timed out has been replaced by  No route to host . This means that when attempting to open connections to the accounting servers, the backup RE immediately decides that the servers are unreachable and does not need to wait for a timeout.