Description

This article explains why an SRX device is reported to be "Out-Of-Sync" in Junos Space after a configuration push from Junos Space to the SRX cluster.

 

Symptoms

The issue is seen when all of the following conditions are met:

  • When the configuration push is initiated from Junos Space to the SRX device

  • When the SRX device has a syslog configuration with "EVENT" mode logging and also has traffic logs enabled

    • Only when "show configuration security log mode"   shows "mode event"
  • Commit message is not found in default-log-messages (the file Junos Space monitors)  (Check to make sure the default-log-messages config file is configured and not disabled. 
    • UI_COMMIT_PROGRESS [junos@xxxxx message="commit complete"] Commit operation in progress: commit complete

When an attempt is made to push configuration changes from Junos Space, the configuration push completes on the SRX device Command Line Interface (CLI), but the SRX cluster goes "Out-Of-Sync" and throws a commit error on Junos Space.

 

Solution

The main reason for this is as follows, if the commit message below is missing from  "show log default-log-messages"

When a commit from Junos Space completes, the SRX device logs a "commit complete" message in syslog.

UI_COMMIT_PROGRESS [junos@xxxxx message="commit complete"] Commit operation in progress: commit complete

But given the current eventd architecture, it is not guaranteed that all logs will reach eventd, particularly when there is excessive logging of messages. And Junos Space relies entirely on eventd messages to synchronize a device. Consequently, if a "commit complete" message is omitted from the SRX device's syslog, Junos Space marks the device as "Out-Of-Sync."

 

There are two ways to resolve this problem:

  1. The SRX cluster must be re-synched in Junos Space. This can be done by modifying the logging mode from "EVENT" to "STREAM," which reduces the pressure on the Routing Engine (RE). When the mode is thus modified, the probability of the RE omitting syslog messages is reduced, leading to a proper sync of the SRX device with Junos Space.

  2. Another workaround that can be implemented is to increase the eventd buffer as follows:

# set event-options max-events-queued 1000

(This is a hidden command, so do ensure that it is typed completely.)

The above command will read the events from sockbuf and delay processing of syslog events. It could be helpful in this case because sockbuf will be cleared quickly by eventd. But this does not guarantee that the issue may not occur again. Under excessive logging, syslog messages may still get dropped.

 

Modification History

2025-02-14: added emphasis on "security log mode event" and notes to check default-log-messages