RE0 was stuck in the following state and we were not able to configure or run show commands on RE0 (Master):
Retry connection attempts ? [yes,no] (yes) could not open user interface connection: management daemon not responding
Could see that several users were stuck in lockf state, as shown in the example below:
10740 root 21 0 480M 41724K lockf 1 0:00 0.00% mgd
11133 root 20 0 480M 41724K lockf 1 0:00 0.00% mgd
11213 root 20 0 480M 41724K lockf 1 0:00 0.00% mgd
11246 root 20 0 480M 41724K lockf 1 0:00 0.00% mgd
Log in to the other RE- RE1, and then enter RE1 shell, and from there, enter RE0 shell using the below commands:
RE1 > start shell user root
login to member0 RE0:
# rsh -T re0
-----------
#ps -Au |grep mgd
You will get an output like the following
root 33835 0.0 0.1 756848 39908 - Is 10:31 0:00.07 mgd: (mgd) (labroot)/dev/pts/0 (mgd) <<<<user is labroot
terminate the process
# kill 33835
Do this for all the users (all the mgd processes), and you should be able to resolve the issue, and enter the CLI.
This errors seems to be because of some commit which did not go through on RE causing a lock on the RE database.
We can verify this by comparing the show system commit. We can also see mgd process being high during issue time causing a database lock.
On Junos and Junos OS Evolved platforms, a situation arises where the mgd (Management Daemon) process fails to release the commit lock, preventing CLI users from modifying the configuration.