This article explains how to to resolve the VPLS with mesh group id stuck in MI state as a chain reaction this will completely fail the l2ald daemon itself.
### Configuration###
labroot@LAB-RouterA-re0> show configuration routing-instancest008_gll_dongctcptgd14
instance-type virtual-switch;
protocols {
vpls {
mesh-group PE {
neighbor 27.68.254.108 {
encapsulation-type ethernet;
ignore-encapsulation-mismatch;
no-vlan-id-validate;
pseudowire-status-tlv;
revert-time 60;
backup-neighbor 27.68.254.107;
}
vpls-id 990001003;
mesh-group SRT {
neighbor 10.18.144.33 {
pseudowire-status-tlv {
hot-standby-vc-on;
traceoptions {
file debug_l3ll size 10m files 10;
flag all detail;
no-tunnel-services;
mtu 9000;
ignore-mtu-mismatch;
mac-flush;
vlans {
t008_gll_dongctcptgd14 {
vlan-id 2710;
### Issue
\* VPLS connection was stucked at MI state *\
labroot@LAB-RouterA-re0> show vpls connections instance t008_gll_dongctcptgd14
Layer-2 VPN connections:
Legend for connection status (St)
EI -- encapsulation invalid NC -- interface encapsulation not CCC/TCC/VPLS
EM -- encapsulation mismatch WE -- interface and instance encaps not same
VC-Dn -- Virtual circuit down NP -- interface hardware not present
CM -- control-word mismatch -> -- only outbound connection is up
CN -- circuit not provisioned <- -- only inbound connection is up
OR -- out of range Up -- operational
OL -- no outgoing label Dn -- down
LD -- local site signaled down CF -- call admission control failure
RD -- remote site signaled down SC -- local and remote site ID collision
LN -- local site not designated LM -- local site ID not minimum designated
RN -- remote site not designated RM -- remote site ID not minimum designated
XX -- unknown connection status IL -- no incoming label
MM -- MTU mismatch MI -- Mesh-Group ID not available
BK -- Backup connection ST -- Standby connection
PF -- Profile parse failure PB -- Profile busy
RS -- remote site standby SN -- Static Neighbor
LB -- Local site not best-site RB -- Remote site not best-site
VM -- VLAN ID mismatch HS -- Hot-standby Connection
Legend for interface status
Up -- operational
Dn -- down
Instance: t008_gll_dongctcptgd14
Mesh-group connections: PE
Neighbor Type St Time last up # Up trans
27.68.254.107(vpls-id 990001003) rmt MI
local PW status code: 0x00000001, Neighbor PW status code: 0x00000000
27.68.254.108(vpls-id 990001003 rmt IL
Mesh-group connections: SRT
10.18.144.33(vpls-id 990001003) rmt OL
\/* VPLS connection was stucked at MI state *\/
2025-07-20 08:21:07.445153965 re0:sysman:15317:TRACE_ERR lltp_err message = "Pending Call Null"
2025-07-20 08:21:07.445155034 re0:sysman:15317:TRACE_ERR lltp_err message = "Error getting status from systemd"
2025-07-20 08:21:07.445556008 re0:sysman:15317:TRACE_ERR lltp_err message = "Pending Call Null"
2025-07-20 08:21:07.445559521 re0:sysman:15317:TRACE_ERR lltp_err message = "Error getting status from systemd"
2025-07-20 08:21:07.445964084 re0:sysman:15317:TRACE_ERR lltp_err message = "Pending Call Null"
2025-07-20 08:21:07.445965423 re0:sysman:15317:TRACE_ERR lltp_err message = "Error getting status from systemd"
2025-07-20 08:21:07.446359511 re0:sysman:15317:TRACE_ERR lltp_err message = "Pending Call Null"
2025-07-20 08:21:07.446360432 re0:sysman:15317:TRACE_ERR lltp_err message = "Error getting status from systemd"
2025-07-20 08:21:07.446786305 re0:sysman:15317:TRACE_ERR lltp_err message = "Pending Call Null"
2025-07-20 08:21:07.446787286 re0:sysman:15317:TRACE_ERR lltp_err message = "Error getting status from systemd"
2025-07-20 08:21:07.447159811 re0:sysman:15317:TRACE_ERR lltp_err message = "Pending Call Null"
2025-07-20 08:21:07.447160746 re0:sysman:15317:TRACE_ERR lltp_err message = "Error getting status from systemd"
2025-07-20 08:21:07.447556024 re0:sysman:15317:TRACE_ERR lltp_err message = "Pending Call Null"
2025-07-20 08:21:07.447556898 re0:sysman:15317:TRACE_ERR lltp_err message = "Error getting status from systemd"
2025-07-20 08:21:07.447964636 re0:sysman:15317:TRACE_ERR lltp_err message = "Pending Call Null"
2025-07-20 08:21:07.447965656 re0:sysman:15317:TRACE_ERR lltp_err message = "Error getting status from systemd"
2025-07-20 08:21:07.448352866 re0:sysman:15317:TRACE_ERR lltp_err message = "Pending Call Null"
2025-07-20 08:21:07.448353735 re0:sysman:15317:TRACE_ERR lltp_err message = "Error getting status from systemd"
2025-07-20 08:21:07.448725312 re0:sysman:15317:TRACE_ERR lltp_err message = "Pending Call Null"
2025-07-20 08:21:07.448726316 re0:sysman:15317:TRACE_ERR lltp_err message = "Error getting status from systemd"
2025-07-20 08:21:07.449101084 re0:sysman:15317:TRACE_ERR lltp_err message = "Pending Call Null"
2025-07-20 08:21:07.449101995 re0:sysman:15317:TRACE_ERR lltp_err message = "Error getting status from systemd"
2025-07-20 08:21:07.449466487 re0:sysman:15317:TRACE_ERR lltp_err message = "Pending Call Null"
2025-07-20 08:21:07.449467399 re0:sysman:15317:TRACE_ERR lltp_err message = "Error getting status from systemd"
<>
2025-07-20 08:19:21.563684930 re0:sysman:15317:TRACE_INFO lltp_info message = "sysmanAppWatcher: path /system/nodes/re0/apps/l2ald type ZOO_DELETED_EVENT"
2025-07-20 08:19:21.563691302 re0:sysman:15317:TRACE_INFO lltp_info message = "Got an App delete l2ald"
2025-07-20 08:19:21.563704128 re0:sysman:15317:TRACE_INFO lltp_info message = "SysManSatellite[re0]: App to be stopped and deleted l2ald service l2ald.service admin_req STOP_UNASSIGN admin_req_id 14418455312132038554"
2025-07-20 08:19:21.563726370 re0:sysman:15317:TRACE_INFO lltp_info message = "SysManExecutor: opStart 61 stopped timer None for l2ald"
2025-07-20 08:19:21.563834086 re0:sysman:15317:TRACE_ERR lltp_err message = "Libsysdbus:Failed to stop service: l2ald.service"
2025-07-20 08:19:21.563837026 re0:sysman:15317:TRACE_INFO lltp_info message = "SysManExecutor: Application stop l2ald.service retry count 0 returned -1"
2025-07-20 08:19:21.563840000 re0:sysman:15317:TRACE_INFO lltp_info message = "SysManExecutor: Retry op Stop for app l2ald.service in 5 secs. retry count 1"
2025-07-20 08:19:21.566341166 re0:sysman:15317:TRACE_INFO lltp_info message = "sysmanAppAdminResponseWatcher: path /system/nodes/re0/apps/l2ald-agent/sysman/adminresponse type ZOO_DELETED_EVENT"
2025-07-20 08:19:21.566347684 re0:sysman:15317:TRACE_INFO lltp_info message = "admin req bit deleted for /system/nodes/re0/apps/l2ald-agent/sysman/adminresponse app l2ald-agent node re0"
2025-07-20 08:19:21.566577653 re0:sysman:15317:TRACE_INFO lltp_info message = "sysmanAppWatcher: path /system/nodes/re0/apps/l2ald-agent type ZOO_DELETED_EVENT"
2025-07-20 08:19:21.566586703 re0:sysman:15317:TRACE_INFO lltp_info message = "Got an App delete l2ald-agent"
2025-07-20 08:19:21.566603740 re0:sysman:15317:TRACE_INFO lltp_info message = "SysManSatellite[re0]: App to be stopped and deleted l2ald-agent service l2ald-agent.service admin_req STOP_UNASSIGN admin_req_id 9453177984122825363"
2025-07-20 08:19:21.566626168 re0:sysman:15317:TRACE_INFO lltp_info message = "SysManExecutor: opStart 61 stopped timer None for l2ald-agent"
2025-07-20 08:19:21.566730848 re0:sysman:15317:TRACE_ERR lltp_err message = "Libsysdbus:Failed to stop service: l2ald-agent.service"
2025-07-20 08:19:21.566733064 re0:sysman:15317:TRACE_INFO lltp_info message = "SysManExecutor: Application stop l2ald-agent.service retry count 0 returned -1"
2025-07-20 08:19:21.566736724 re0:sysman:15317:TRACE_INFO lltp_info message = "SysManExecutor: Retry op Stop for app l2ald-agent.service in 5 secs. retry count 1"
2025-07-20 08:19:26.564676187 re0:sysman:15317:TRACE_INFO lltp_info message = "SysManExecutor: eventHandler 373 SysManExecutor::eventHandler Pending op Stop timeout for l2ald"
2025-07-20 08:19:26.564743080 re0:sysman:15317:TRACE_ERR lltp_err message = "Libsysdbus:Failed to stop service: l2ald.service"
2025-07-20 08:19:26.564744976 re0:sysman:15317:TRACE_INFO lltp_info message = "SysManExecutor: Application stop l2ald.service retry count 1 returned -1"
2025-07-20 08:19:26.564747530 re0:sysman:15317:TRACE_INFO lltp_info message = "SysManExecutor: Retry op Stop for app l2ald.service in 5 secs. retry count 2"
2025-07-20 08:19:26.566857746 re0:sysman:15317:TRACE_INFO lltp_info message = "SysManExecutor: eventHandler 373 SysManExecutor::eventHandler Pending op Stop timeout for l2ald-agent"
2025-07-20 08:19:26.567099368 re0:sysman:15317:TRACE_ERR lltp_err message = "Libsysdbus:Failed to stop service: l2ald-agent.service"
2025-07-20 08:19:26.567110713 re0:sysman:15317:TRACE_INFO lltp_info message = "SysManExecutor: Application stop l2ald-agent.service retry count 1 returned -1"
2025-07-20 08:19:26.567121553 re0:sysman:15317:TRACE_INFO lltp_info message = "SysManExecutor: Retry op Stop for app l2ald-agent.service in 5 secs. retry count 2"
2025-07-20 08:19:31.565510851 re0:sysman:15317:TRACE_INFO lltp_info message = "SysManExecutor: eventHandler 373 SysManExecutor::eventHandler Pending op Stop timeout for l2ald"
2025-07-20 08:19:31.565666987 re0:sysman:15317:TRACE_ERR lltp_err message = "Libsysdbus:Failed to stop service: l2ald.service"
2025-07-20 08:19:31.565671196 re0:sysman:15317:TRACE_INFO lltp_info message = "SysManExecutor: Application stop l2ald.service retry count 2 returned -1"
2025-07-20 08:19:31.565676600 re0:sysman:15317:TRACE_INFO lltp_info message = "SysManExecutor: Retry op Stop for app l2ald.service in 5 secs. retry count 3"
2025-07-20 08:19:31.566746027 re0:sysman:15317:TRACE_INFO lltp_info message = "SysManExecutor: eventHandler 373 SysManExecutor::eventHandler Pending op Stop timeout for l2ald-agent"
2025-07-20 08:19:31.566868011 re0:sysman:15317:TRACE_ERR lltp_err message = "Libsysdbus:Failed to stop service: l2ald-agent.service"
2025-07-20 08:19:31.566870870 re0:sysman:15317:TRACE_INFO lltp_info message = "SysManExecutor: Application stop l2ald-agent.service retry count 2 returned -1"
Once you see the above configuration and syslogs are matching then we can confirm that we are hitting this PR1866988.
From the sysman trace logs we can see that the DBUS is wedged, Systemd is not responding to any of the requests from sysman via sys DBUS.
(like systemctl start l2ald or systemctl reset-failed l2ald etc)
Clear the l2ald fail alarm by following the below procedure without going with the complete system reboot:
From the shell
1. #kill -s USR1 1
2. #kill -s TERM 1
wait for 5 seconds
3. #systemctl restart dbus
cli>request system application app l2ald-agent restart node re0
Basically any app failed where sysman fails to make a connection will be hitting this PR and need to follow the WA mentioned in my previous mail.
Its just a work around better to upgrade for the fix once its available.
Below is the list of fixed junos releases.
Resolved-In
evo:24.2R2-S2-EVO evo:24.4R2-EVO evo:25.2R1-EVO evo:25.2R2-EVO evo:25.3R1-EVO evo:25.4R1-EVO