Description

Autoresync is not working, post SD update job is performed.

Symptoms

Update job might get failed in SD with error "statement creation failed"
Issue might get resolved after a manual resync with the device, but it might reappear after next update.

Solution

  • When a commmit happens on the Device, you will receive the following message in Space.
  • Check for below log in server.log. (get the device id from LogicalDevice table)
WARN [net.juniper.jmp.utils.DeviceStatusNotificationUtil] (SyslogWorkerPool-27) Publish 'Device Status Change' notification for device 32789 
WARN [net.juniper.jmp.cems.inventoryManagerImpl.JmpInventoryCommands] (SyslogWorkerPool-21) Auto resync scheduled for device:10.85.241.23 id:32789
  • If you don't see this message then the device is not informing the space about the commit change and the auto-resync workflow is not working.
  • Check from the device side if the required config is in place.

Example:

set system syslog file default-log-messages any info
set system syslog file default-log-messages match "(requested 'commit' operation)|(requested 'commit synchronize' operation)|(copying configuration to juniper.save)|(commit complete)|ifAdminStatus|(FRU power)|(FRU removal)|(FRU insertion)|(link UP)|transitioned|Transferred|transfer-file|(license add)|(license delete)|(package -X update)|(package -X delete)|(FRU Online)|(FRU Offline)|(plugged in)|(unplugged)|GRES"
set system syslog file default-log-messages structured-data
  • If configurations are correct then check if the device is actually sending message to space via netconf.
  • From the SRX side enable traceoptions for netconf and tail netconf-ops.log and perform a plain commit in a new shell window. This step will let us know if the device is sending the commit change message to Space.
  • Note: Please check with the customer if they are ok to enable netconf debug before proceeding. If you are unsure about below debug config then get help from SRX team.
set system services netconf ssh
set system services netconf traceoptions file netconf-ops.log
set system services netconf traceoptions file size 5m
set system services netconf traceoptions file files 10
set system services netconf traceoptions file world-readable
set system services netconf traceoptions flag all
  • Expected NETCONF output below when a commit happens on the device. If this output is not seen then involve an SRX engineer to troubleshoot this from SRX side.
  • In some cases the eventd process in SRX might get stuck with high cpu, due to which the SRX is unable to send messages to space over netconf. Restarting the service might help in fixing the issue.

tail -f /var/log/netconf-ops.log

+------------------------------------------------------------+

May 12 21:59:38 [NETCONF] Started tracing session: 95326

May 12 21:59:38 [NETCONF] - [95326] Outgoing: <!-- No zombies were killed during the creation of this user interface -->

May 12 21:59:38 [NETCONF] - [95326] Outgoing: <!-- user labroot, class j-super-user -->

May 12 21:59:38 [NETCONF] - [95326] Outgoing: <hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">

May 12 21:59:38 [NETCONF] - [95326] Outgoing:  <capabilities>

  <capability>urn:ietf:params:netconf:base:1.0</capability>

  <capability>urn:ietf:params:netconf:capability:candidate:1.0</capability>

  <capability>urn:ietf:params:netconf:capability:confirmed-commit:1.0</capability>

  <capability>urn:ietf:params:netconf:capability:validate:1.0</capability>

  <capability>urn:ietf:params:netconf:capability:url:1.0?scheme=http,ftp,file</capability>

  <capability>urn:ietf:params:xml:ns:netconf:base:1.0</capability>

  <capability>urn:ietf:params:xml:ns:netconf:capability:candidate:1.0</capability>

  <capability>urn:ietf:params:xml:ns:netconf:capability:confirmed-commit:1.0</capability>

  <capability>urn:ietf:params:xml:ns:netconf:capability:validate:1.0</capability>

  <capability>urn:ietf:params:xml:ns:netconf:capability:url:1.0?scheme=http,ftp,file</capability>

  <capability>urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring</capability>

  <capability>http://xml.juniper.net/netconf/junos/1.0</capability>

  <capability>http://xml.juniper.net/dmi/system/1.0</capability>

May 12 21:59:38 [NETCONF] - [95326] Outgoing:  </capabilities>

May 12 21:59:38 [NETCONF] - [95326] Outgoing:  <session-id>95326</session-id>

May 12 21:59:38 [NETCONF] - [95326] Outgoing: </hello>

May 12 21:59:38 [NETCONF] - [95326] Outgoing: ]]>]]>

May 12 21:59:38 [NETCONF] - [95326] Incoming: <?xml version="1.0" encoding="UTF-8"?><hello><capabilities><capability>urn:ietf:params:xml:ns:netconf:base:1.0</capability></capabilities></hello>]]>]]>

May 12 21:59:38 [NETCONF] - [95326] Incoming: <?xml version="1.0" encoding="UTF-8"?><rpc message-id="1"><get-log><filename>inventory</filename></get-log></rpc>

]]>]]>

May 12 21:59:38 [NETCONF] - [95326] Outgoing: <rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:junos="http://xml.juniper.net/junos/20.4R0/junos" message-id="1">

May 12 21:59:38 [NETCONF] - [95326] Outgoing: <file-content filename="inventory" junos:seconds="1652372978" filesize="4113" encoding="text">

May 12 21:59:38 [NETCONF] - [95326] Outgoing: May 2 18:48:26 CHASSISD release 20.4R3-S2.6 built by builder on 2022-02-06 18:29:06 UTC

--

--

May 12 21:59:38 [NETCONF] - [95326] Outgoing: </file-content>

May 12 21:59:38 [NETCONF] - [95326] Outgoing: </rpc-reply>

May 12 21:59:38 [NETCONF] - [95326] Outgoing: ]]>]]>

May 12 21:59:38 [NETCONF] - [95326] Incoming: <rpc><close-session/></rpc>]]>]]>

May 12 21:59:38 [NETCONF] - [95326] Outgoing: <rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:junos="http://xml.juniper.net/junos/20.4R0/junos">

May 12 21:59:38 [NETCONF] - [95326] Outgoing: <ok/>

May 12 21:59:38 [NETCONF] - [95326] Outgoing: </rpc-reply>

May 12 21:59:38 [NETCONF] - [95326] Outgoing: ]]>]]>

May 12 21:59:38 [NETCONF] - [95326] Outgoing: <!-- session end at 2022-05-12 21:59:38 IST -->

+------------------------------------------------------------+

Modification History

2024-12-17 : Article Created