Description

Error could be affecting the RE information from CLI show commands on PTX series.

Symptoms

some show commands no showing RE information, and error message reported, as following example:

re0> show chassis routing-engine

 

Routing Engine status:

 

Slot 0:

 

Current state Master

 

Election priority Master

 

Temperature 38 degrees C / 100 degrees F

 

CPU temperature 38 degrees C / 100 degrees F

 

DRAM 34464 MB (65536 MB installed)

 

Memory utilization 47 percent

 

5 sec CPU utilization:

 

User 2 percent

 

Background 0 percent

 

Kernel 1 percent

 

Interrupt 0 percent

 

Idle 96 percent

 

1 min CPU utilization:

 

User 2 percent

 

Background 0 percent

 

Kernel 1 percent

 

Interrupt 0 percent

 

Idle 96 percent

 

5 min CPU utilization:

 

User 2 percent

 

Background 0 percent

 

Kernel 1 percent

 

Interrupt 0 percent

 

Idle 96 percent

 

15 min CPU utilization:

 

User 2 percent

 

Background 0 percent

 

Kernel 1 percent

 

Interrupt 0 percent

 

Idle 96 percent

 

Model JNP10K-RE1-E

 

Serial ID BUILTIN

 

Start time 2023-07-27 06:16:00 UTC

 

Uptime 222 days, 15 hours, 37 minutes, 12 seconds

 

Load averages: 1 minute 5 minute 15 minute

 

0.79 0.90 0.97

 

Last reboot reason software reboot

 

error: communication failure with /re1/hwdre/stack1

 

1. The producer app is either down or unresponsive

 

Run "show system processes node <node> | grep <app>" to check if app is running

 

Run "show system application app <app>" to check the state of the app

 

2. If there is a change in mastership recently, please wait for the switchover to complete for the app to be online

 

Run "request chassis routing-engine master switch check" to check the status of switchover

Solution

= root cause =

There is a possible dead-lock condition when a multi-threaded application indirectly invokes a fork() system call via popen() and system().

The SI database mutex may be held at the time of fork() by a different thread in the process, and the new process may block forever attempting to obtain the SI database mutex as a result.

 

= fix =

As these APIs may internally fork(), we must ensure that the SI DB lock is held in the thread that will be forking, otherwise we may have deadlock in the child process.

 

  • Like current error is present on RE1, you log in RE1 (no masterswitch only log in on RE1)
  • Run this command: request system application restart app hwdre node re1
  • This command mention may affect traffic, but I noticed the restart was extremely quick and I checked other cases reference and mention no affect the traffic, However you could consider to apply it in a short MW. But is your decision.

 

labroot@e04-30> request system application restart app hwdre node re0

This may affect traffic in the system. Proceed ? [yes,no] (no) yes

 

  • Then you could verify the hwdre status with this command: show system applications app hwdre node re1
  • From offline, stand by to online-ready active state, EX:

PTX10k8-47-re0> show system applications app hwdre node re0

Applications Information:

Application       : hwdre

Node          : re0

App State        : offline

App Weight       : 0

Object Producer details

Producer ID       : 50331902

 Epoch ID        : 0

 Production Topic    : /Root/hwdre/6231521938600547355

 Producer State     : standby

Producer ID       : 50332514

 Epoch ID        : 0

 Production Topic    : /Root/re0/hwdre/7442401238514052515

 Producer State     : standby

 

labroot@e04-30> show system applications app hwdre node re0

Applications Information:

Application       : hwdre

Node          : re0

App State        : online ready

App Weight       : 2

App Zookeeper Session  : 1000000c58a0070

Object Producer details

Producer ID       : 50332072

 Epoch ID        : 108

 Production Topic    : /Root/re0/hwdre/4181993935048745161

 Producer State     : active

Producer ID       : 50331891

 Epoch ID        : 109

 Production Topic    : /Root/hwdre/12929773520120618483

 Producer State     : active

 

After restart the app hwdre the issue should be resolve, you will be able to see RE1 information.

Modification History

2024-03-13 : Article Created