This article discusses an issue on QFX5200, QFX5210, QFX5110, and QFX5120 platforms where the "show version" command returns the error "timeout waiting for response from localre" or displays [] instead of the host software information. Additionally, the FPCs may remain offline, the "show chassis hardware" output may exclude the FPC and FRUs, and the device may reboot due to a watchdog event.

Symptom #1 is the primary indicator of the issue.

It may occur independently or alongside symptoms #2 and/or #3.

Symptom #1: 

The "show version" command times out and returns an error.

root@qfx5200> show version | no-more
localre:
--------------------------------------------------------------------------
Hostname: qfx5200
Model: qfx5200-32c-32q
Junos: 22.2R3-S3.18 flex
JUNOS OS Kernel 64-bit FLEX [20231130.89ecb3b_builder_stable_12_222]
>>>>>>>>>>>>>>> SNIPPED FOR BREVITY <<<<<<<<<<<<<<<
JUNOS dsa dsa [22.2R3-S3.18]
JUNOS Host Software [3.14.52-rt50-WR7.0.0.9_ovp:3.1.0]
error: timeout waiting for response from localre

[OR]

root@qfx5200> show version | no-more
localre:
--------------------------------------------------------------------------
Hostname: qfx5200
Model: qfx5200-32c-32q
Junos: 22.2R3-S3.18 flex
JUNOS OS Kernel 64-bit FLEX [20231130.89ecb3b_builder_stable_12_222]
>>>>>>>>>>>>>>> SNIPPED FOR BREVITY <<<<<<<<<<<<<<<
JUNOS dsa dsa [22.2R3-S3.18]
JUNOS Host Software [3.14.52-rt50-WR7.0.0.9_ovp:3.1.0]
[]
[]
[]
[]
[]
[]
Junos for Automation Enhancement

Symptom #2: 

• The FPC does not come up. 
• The "show chassis hardware" command does not list the FPC and FRUs.

root> show chassis hardware
Hardware inventory:
Item       Version Part number Serial number   Description
Chassis                        WH4824360113
Pseudo CB 0

Symptom #3: 

• The device may experience a random watchdog reboot.
• The reboot reason will be recorded as 0x80000:CPU Reset in "show chassis routing-engine".

root@qfx5200> 
watchdog watchdog0: watchdog did not stop!
RE-FPGA-DRV: reboot notifier called with 0x0001
RE-FPGA-DRV: Please standby while rebooting.
Warm reboot initiated ...
..
..

RPM database corruption in the DCPFE container causes this issue.

The “show version” command will initiate an RPM query in the DCPFE container to retrieve package information. If the RPM database is corrupted, the query will fail. This corruption may prevent the DCPFE and/or LCMD processes from starting, which would cause the FPC to fail to initialize. Consequently, the FPC/FRU will not appear in the “show chassis hardware” output. The corrupted RPM database may also cause RPM queries to get stuck in a loop, increasing CPU usage. This can deplete system resources, affecting other processes, including those responsible for “patting” the watchdog, which could lead to a watchdog reboot.

RPM database corruption in the DCPFE container can occur if the switch is abruptly halted during bootup or if the system is halted using the “request system halt” command while an RPM query is in progress. For instance, an RPM query may run when executing the “show version” command or during certain system operations. While the “request system halt” command gracefully shuts down Junos, it causes the Host OS to shut down abruptly.

How do I confirm RPMDB in the DCPFE container is corrupted?
RPM database corruption can be confirmed if the following command returns this error: 
> request app-engine host-cmd "/usr/bin/lxc-attach -n dcpfe0 -- /usr/sbin/jlxc_pkg_info"
error: cannot open Packages database in /var/lib/rpm
rpmdb: BDB1508 Unable to allocate thread control block

A healthy switch will return the following output [version may differ]: 
> request app-engine host-cmd "/usr/bin/lxc-attach -n dcpfe0 -- /usr/sbin/jlxc_pkg_info"
JUNOS Host qfx-5e platform package [22.4R3-S6.5]
JUNOS Host qfx-5e base package [22.4R3-S6.5]
JUNOS Host qfx-5e data-plane package [22.4R3-S6.5]

Recovery: 
The corrupted database can be removed and rebuilt using the following commands:
> request app-engine host-cmd "/usr/bin/lxc-attach -n dcpfe0 -- rm -rf /var/lib/rpm/__db.00*"
> request app-engine host-cmd "/usr/bin/lxc-attach -n dcpfe0 -- rpm -vv --rebuilddb"

Alternatively, upgrading the switch using "request system software add force-host <>" or a format install can recover this issue. 

Software Enhancement/Fix: 
A software enhancement to automatically detect and resolve RPM database corruption at the DCPFE container level has been introduced through PR 1850037. Refer to the PR or contact JTAC for a list of releases that include the fix.