Need to investigate to find the root cause for why the RE went to offline state.
error: Failed to get RE count
warning: 'synchronize' ignored, other routing engine is offline
The logs in symptoms are usually seen soon after the RE switchover, if a user tries to commit the configuration, mgd is unable to figure out the RE
But in this case, there were no such commits or no other related logs for the RE going to offline state were seen.
Engineering identified that,
1. CMDD is executing "show system nodes" from mgd to get number of REs
2. CMDD forwards command to sysman on RE1
3. sysman on RE1 has error messages related to systemd
Also there is no platform level link state change on the inter-RE link.
DBUS errors could be seen, which is related to a PR were DBUS went into bad state for some time but recovered after couple of hours, but sysman was still having stale old connection to D-BUS.
As per engineering, the issue is fixed via PR1866988.
Due to the internal process communication failure, Junos OS Evolved system might fail to response to system commands. For example, the CLI commands starting from "request system" or "request node".
Logs:
2025-04-21 14:12:17.900391181 re0:cmdd:15406:TRACE_INFO lltp_info message = "session id 92951: complete XML document from mgd"
2025-04-21 14:12:17.900397814 re0:cmdd:15406:TRACE_INFO lltp_info message = "session id 92951: command string: 'show system nodes'"
|
2025-04-21 14:12:17.900468799 re0:cmdd:15406:TRACE_INFO lltp_info message = "DdxClientApp::Connect: app = /re1/sysman/, ip = 128.0.0.5, port = 53161, ipc_client = 0x7fec0a73bd00, stream = 0x7fec073de080, topic/key = /junos/system-nodes-info-entry"
2025-04-21 14:12:17.899643322 re0:sysman:15348:TRACE_WARNING lltp_warning message = "DdxServerConn::rcvDdxRequestMsg: received cancel request, stream not present, drop the request, this = 0x7f0793a60800, stream_id = 155268"
2025-04-21 14:12:19.486771565 re0:sysman:15348:TRACE_WARNING lltp_warning message = "DdxServerConn::rcvDdxRequestMsg: received cancel request, stream not present, drop the request, this = 0x7f0793a60800, stream_id = 155270"
2025-04-21 14:12:25.052446411 re1:sysman:27248:TRACE_ERR lltp_err message = "There was an error in response for service: ppmdagent.service"
2025-04-21 14:12:25.052449679 re1:sysman:27248:TRACE_ERR lltp_err message = "Response message error: Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)"
2025-04-21 14:12:50.078039476 re1:sysman:27248:TRACE_ERR lltp_err message = "Response message error: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken."
2025-06-25 : Article Created
2025-08-01 : Hyper linked with PR1866988