Summary:
When the backup re (re1) is swapped out, the output of the "show chassis routing-engine" on the master re (re0) shows re1 in a state of "present" with re0 state as master.
On re1, the one that was swapped, it shows the state of the master re (re0) as "present" and its own state as "backup". There is no "master" in this output.
The only way we have been able to rectify the fault is to reload the master re.
=========================
More details:
The setup that is being used is an MX304 with dual REs and a single LMIC, running the 24.4R1-S2.9 code.
We have been undertaking testing of the MX304. One of the tests is to simulate an RE hot swap. When the "swapped" re comes back online, the master re (re0) is unable to communicate with the standby re (re1) and shows re1 having a state of present. I have also noticed that the bme interfaces is in an up/down state during this period.
The only way I have been able to restore communications between re0 and re1 is to reload re0, which was the master at the time.
The test consists of
- Checking the system was in a healthy state
- Stopping the backup RE, re1 in the test cases.
- When re1 is down, removing the re from the chassis
- Waiting 2 - 5 minutes.
- Re-installing re1 into the chassis
- recording the bootup process.
The documentation is unclear as to which exact commands should be used, with step 1 just stating to "Take the host subsystem offline."
https://www.juniper.net/documentation/us/en/hardware/mx304/topics/topic-map/mx304-maintaining-rcb.html
I have run the test multiple times, with multiple variations, the following tests resulted in the RE communication issue.
- Run the test and stopped re1 with the command "request vmhost power-off re1" from re0
- Run the test and stopped re1 with the command "request vmhost power-off re1" from re1
- Run the test and stopped re1 with the command "request vmhost halt re1" from re0
- Run the test and stopped re1 with the command "request vmhost halt re1" from re1
- Disabling GRES and run the test and stopped re1 with the command "request vmhost power-off re1" from re0
- Disabling GRES run the test and stopped re1 with the command "request vmhost power-off re1" from re1
- Disabling GRES run the test and stopped re1 with the command "request vmhost halt re1" from re0
- Disabling GRES run the test and stopped re1 with the command "request vmhost halt re1" from re1
- At this point, I zeroized the system, both res, and put a very basic config on. Just hostname, deleted autoupgrade, set usernames and password.
- With the basic configuration, Run the test and stopped re1 with the command "request vmhost power-off re1" from re0
- With the basic configuration, Run the test and stopped re1 with the command "request vmhost halt re1" from re0
- With the basic configuration, Run the test and stopped re1 with the command "request vmhost halt re1" from re1
I have also tried rebooting the re, and that works successfully, when the re comes back both re0 and re1 communications works.
- With the basic configuration, execute the "request vmhost reboot re1" from re0
- With the basic configuration, execute the "request vmhost reboot routing-engine local" from re1
user@CDM-JMX304-001> show chassis routing-engine
Routing Engine status:
Slot 0:
Current state Master
Slot 1:
Current state Present
user@CDM-JMX304-001>
user@JMX304-001> show chassis alarms
2 alarms currently active
Alarm time Class Description
2025-06-18 15:06:09 GMT Minor Loss of communication with Backup RE
2025-06-18 15:03:08 GMT Major Host 0 bme1 : Ethernet Link to other RE Down
user@JMX304-001> request routing-engine login other-routing-engine
connect to address 128.0.0.5: No route to host
Trying 10.0.0.5...
re1: Network is down
{master}
user@JMX304-001> show vmhost version
Current root details, Device sda, Label: jrootp_P, Partition: sda3,
Current boot disk: Primary
Current root set: p
UEFI Version: CICL_P_BUG1_00.23.01
Primary Disk, Upgrade Time: Tue Jun 17 11:31:17 AEST 2025
Version: set p
VMHost Version: 10.3251
VMHost Root: vmhost-x86_64-24.4R1-S2-20250219_2102_builder
VMHost Core: vmhost-core-x86-64-24.4R1-S2.9
kernel: 5.15.161-rt76-LTS22
Junos Disk: junos-install-mx-x86-64-24.4R1-S2.9
Version: set b
Jul 1 08:12:38 CHASSISD_LINK_RE_ERROR_RECOVER: RE_TO_ORE iface recovery: bme1 is down <<<< Chassisd is try to recover the interface, yet it is not coming up
Jul 1 08:12:40 rcb_temp_sensor_all_status_update: skip for FRU : 1
Jul 1 08:12:43 rcb_temp_sensor_all_status_update: skip for FRU : 1
Jul 1 08:12:43 tvp_cb_periodic: skip for FRU : 1
Jul 1 08:12:43 CHASSISD_LINK_RE_ERROR_RECOVER: RE_TO_ORE iface recovery: bme1 is down
Jul 1 08:12:46 rcb_temp_sensor_all_status_update: skip for FRU : 1
Jul 1 08:12:48 tvp_cb_periodic: skip for FRU : 1
Jul 1 08:12:48 CHASSISD_LINK_RE_ERROR_RECOVER: RE_TO_ORE iface recovery: bme1 is down
Jul 1 08:12:49 rcb_temp_sensor_all_status_update: skip for FRU : 1
Jul 1 08:12:52 rcb_temp_sensor_all_status_update: skip for FRU : 1
Jul 1 08:12:53 tvp_cb_periodic: skip for FRU : 1
Jul 1 08:12:53 CHASSISD_LINK_RE_ERROR_RECOVER: RE_TO_ORE iface recovery: bme1 is down
Jul 1 08:12:55 rcb_temp_sensor_all_status_update: skip for FRU : 1
Jul 1 08:12:58 rcb_temp_sensor_all_status_update: skip for FRU : 1
Jul 1 08:12:58 tvp_cb_periodic: skip for FRU : 1
The reported issue is easily replicable and its considered as a bug and more details are shared below
Please find the external PR link below :
[MX304] Jack-out and Jack-in standby RE results in communication loss between REs
https://prsearch.juniper.net/problemreport/PR1891577
workaround:
Bounce bme1 interface from VMhost :
root@-node:~# virsh domif-setlink vjunos tap2 down
Device updated successfully
root@-node:~# virsh domif-setlink vjunos tap2 up