Description

This article explains why restaring the hwdre process may trigger RE switchover based on below PTX router running EVO code.

Model: ptx10008

Junos: 21.4R3-S3.6-EVO

Symptoms

On customer router we observed below log messages after we initiated the workaround to restart hwdre process since the EVO code did not have the fix committed to this release:

2024-08-23T12:42:23.938Z PTX-re0 systemd 1 - - hwdre.service: State 'stop-sigterm' timed out. Killing.

2024-08-23T12:42:24.207Z PTX-re0 systemd 1 - - hwdre.service: Main process exited, code=killed, status=9/KILL

2024-08-23T12:42:24.219Z PTX-re0 sysman 17060 SYSTEM_APP_OFFLINE_EVENT [[email protected] message="SYSTEM_APP_OFFLINE_EVENT: App went offline re0-hwdre"] App went offline re0-hwdre

2024-08-23T12:42:26.626Z PTX-re0 systemd 1 - - Stopped "Hardware Daemon on RE".

2024-08-23T12:42:26.626Z PTX-re0 systemd 1 - - hwdre.service: Unit entered failed state.

2024-08-23T12:42:26.627Z PTX-re0 systemd 1 - - hwdre.service: Triggering OnFailure= dependencies.

2024-08-23T12:42:26.627Z PTX-re0 systemd 1 - - hwdre.service: Failed with result 'signal'.

2024-08-23T12:42:26.653Z PTX-re0 sysman 17060 SYSTEM_APP_FAILED_EVENT [[email protected] message="SYSTEM_APP_FAILED_EVENT: App has failed re0-hwdre"] App has failed re0-hwdre

2024-08-23T12:42:26.653Z PTX-re0 systemd 1 - - Started Failure Handler Service.

2024-08-23T12:42:26.654Z PTX-re0 emfd-fpa 19657 - - EMF_EVO_ALARM_SET: Alarm set: APP color=red, class=CHASSIS, reason=Application hwdre fail on node Re0

2024-08-23T12:42:26.654Z PTX-re0 systemd 1 - - Started "Hardware Daemon on RE".

2024-08-23T12:42:26.654Z PTX-re0 emfd-fpa 19657 - - EMF_FPA_ALARM_REP: RaiseAlarm: Alarm(Location: /Chassis[0]/Re[0] Module: sysman Object: hwdre:0 Error: 2) reported

 

2024-08-23T12:42:31.133Z PTX-re0 jinsightd 11723 - - JRESIL_MAINPROC_ERRITEM: Central Resiliency Entity (jResil) is processing an error with uri: /re/0/emfd-fpa/0/hwdre:0/0/sysman/0/application_major_failure

2024-08-23T12:42:32.268Z PTX-re0 jinsightd 11723 - - JRESIL_MAINPROC_ACTION: Central Resiliency Entity (jResil) executed action: ReSwitchover

Solution

We checked with developer that Switchover is an expected behavior when hwdre application fails on the primary RE. When “systemctl restart hwdre” is executed on the primary RE and hwdre application is having communication failure, hwdre app did not exit on sigterm and systemd had to kill it moving the application to failed state before restarting it. Jinsghtd triggered the SWO due to hwdre failure.

 

2024-08-23T12:42:24.207Z PTX-re0 systemd 1 - - hwdre.service: Main process exited, code=killed, status=9/KILL <<<<< hwdre app didnt exit on sigterm and systemd had to kill it, moving app to failed state

2024-08-23T12:42:32.268Z PTX-re0 jinsightd 11723 - - JRESIL_MAINPROC_ACTION: Central Resiliency Entity (jResil) executed action: ReSwitchover <<<<<< jinsightd is triggering SWO due to hwdre failure

 

When communication failure is observed with re/hwdre error on the primary RE, kindly apply the workaround:

  • Schedule a Maintenance window.
  • Then switch to Secondary RE (RE1) as the primary RE, then restart the hwdre process on Primary RE (RE0). After the hwdre process restarts and there are no more communication failure messages, you could switch back the RE0 as primary.

 

To restart hwdre process you will need to login to the root shell-

> start shell user root

> systemctl restart hwdre

Modification History

2024-09-23 : Initial Publication