On Junos and Junos OS Evolved platforms, a situation arises where the mgd (Management Daemon) process fails to release the commit lock, preventing CLI users from modifying the configuration. This issue becomes more critical when the maximum edit sessions limit is reached in this state. Although there is no direct service impact, CLI users encounter an error message: "Could not login to the database, maximum edit sessions limit."
Continuous occurrence of logs indicating the MGD process reaching the ceiling limit on the backup routing engine.
Numerous MGD processes running on the routing engine.
MGD processes get stuck in the lockf state during commit operations, causing a cascade effect on subsequent sessions.
Inability for CLI users to enter configuration mode.
Error message: "Could not login to the database, maximum edit sessions limit."
Logs: Continuous logs indicating a process reaching the ceiling limit on the backup routing engine:
jlaunchd[18262]: System reaching processes ceiling low watermark: Contact the system administrator to clean up unnecessary processes or increase maxproc ceiling. jlaunchd[18262]: System reaching processes ceiling high watermark: Contact the system administrator to clean up unnecessary processes or increase maxproc ceiling. Further process fork requests may be denied.
A log of the MGD session in the lockf state:
user@device> show system processes extensive | match mgd 35462 root 103 0 735M 46M CPU0 0 3171.1 100.00% mgd 57419 root 20 0 734M 45M select 0 0:16 0.00% mgd ... 63524 root 38 0 734M 45M lockf 2 0:00 0.00% mgd 88773 root 22 0 734M 45M lockf 0 0:00 0.00% mgd ... 49788 root 39 0 734M 45M lockf 0 0:00 0.00% mgd 65397 root 22 0 734M 45M lockf 2 0:00 0.00% mgd ...
System recovery logs after hitting the ceiling:
Message from syslogd@user at Dec 19 11:44:02 2023 ... user last message repeated 3 times Message from syslogd@user at Dec 19 11:44:04 2023 ... user re1 2023 user other-RE: jlaunchd[18262]: Proctable recovery killed process pid(205), ppid(57419), name(mgd), dsnd(0) ... labroot jlaunchd[18262]: System processes overflow: Recovered the system by killing malefactor processes. Contact the system administrator to verify the system state.
To address the issue of CLI users being unable to access configuration mode on Junos and Junos OS Evolved platforms, follow these steps:
Identify and Kill MGD Processes: Determine the problematic MGD process that is stuck in the lockf state and preventing other sessions from proceeding. Use the following command to kill all MGD sessions in the system:
ps wuax | grep mgd | awk '{print $2}' | xargs kill -9
This step helps release the commit lock and allows recovery from the stuck state.
This issue is related to a known problem tracked via PR 1761939