Description

The LCMD (Linux Chassis Management Daemon) is present on devices that have NGRE (Next Generation Routing Engine) running on the VMHost (Linux host). This will help the chassis daemon running in Junos OS to interact with the rest of the hardware components.

This article explains how to overcome LCMD application failure on Junos NGRE.

Symptoms

  1. RE Mastership Switchover
  2. Other hardware component alarms along with LCMD alarm

Example Alarms:

root@host> show chassis alarms no-forwarding

14 alarms currently active
Alarm time               Class  Description
2020-06-02 14:21:57 UTC  Minor  VMHost RE 1 host lcmd Application failed
2020-06-02 14:19:19 UTC  Minor  PEM 5 Feed 2 switch OFF but input connected   <-- List of hardware alarms for PEM accompanying LCMD alarm
2020-06-02 14:19:19 UTC  Minor  PEM 5 Feed 1 switch OFF but input connected
2020-06-02 14:19:19 UTC  Minor  PEM 4 Feed 2 switch OFF but input connected
2020-06-02 14:19:19 UTC  Minor  PEM 4 Feed 1 switch OFF but input connected
2020-06-02 14:19:19 UTC  Minor  PEM 3 Feed 2 switch OFF but input connected
2020-06-02 14:19:19 UTC  Minor  PEM 3 Feed 1 switch OFF but input connected
2020-06-02 14:19:19 UTC  Minor  PEM 2 Feed 2 switch OFF but input connected

Solution

Precautionary Measure:

This feature will switch mastership to the other-routing engine once LCMD connectivity between VM and RE is broken. To enable this feature, including the 'on-loss-of-vm-host-connection' statement at the '[edit chassis redundancy failover]' hierarchy level:

[edit chassis redundancy failover]
on-loss-of-vm-host-connection;


If the failover triggers after LCMD issue, you can see the 'Backup RE Active ' alarm in the active alarm list.

Verify if there is a crash file for LCMD:

user@host> show app-engine crash 
Crash Info
     ==========
     total 13480
     -rw-r--r-- 1 root root  172958 May 14 23:08 localhost.lcmd.23743.1492946195.core.tgz

Re-initiate LCMD process to initiate chassis communication:

user@host> start shell user root 
Password:

root@host:/var/home/user # vhclient -s
Last login: Thu Apr 24 16:39:59 AEDT 2020 from router on pts/1

Make a note of process ID:

root@host-node:~# ps aux | grep lcmd

root      2542   7.8  0.0 297024 10568 ?     SLsl 16:43   0:06 lcmd   <-- PID is 2542

Restart the process by initiating kill:

root@host-node:~# kill -9 <PID>

Example:

root@host-node:~# kill -9 2542

Verify LCMD re-initiates with a new PID:

root@host-node:~# ps aux | grep lcmd
root      2873 30.9  0.0 297024 10476 ?     SLsl 16:44   0:05 lcmd    <-- New LCMD  

Alternatively, you can also verify the PID of LCMD process with the command below in configuration mode:

root@host> show vmhost processes

Once LCMD process has been re-initiated, check if the alarms disappear.  

If the alarms still persist, restart the RE in a safe maintenance window.

To obtain host logs for Root Cause Analysis, refer to KB31910 - How to collect /var/log files from Next Generation Routing Engine (NG-RE) [juniper.net]

Once you obtain the host logs, navigate to LCMD files and check the logs for detailed RCA.

NOTE:  For detailed RCA from LCMD logs and crash file, suspicion of RE hardware issue, please contact your JTAC Representative.

WARNING: Please ensure GRES and NSR is enabled for mastership switchover precautionary measure.

Modification History

2024-05-14: Article updated with EX9253

Related Information