Description

This article explains how to troubleshoot when a QFX5100 switch (Standalone or in a VC-setup) loses in-band access and all the ports go down.

Symptoms

The device might become unstable and might or might not be able to generate logs during the outage. You may notice the below logs in console output:

ad1: FAILURE - WRITE_DMA status=41<READY,ERROR> error=4<ABORTED> LBA=20031

ad1: FAILURE - WRITE_DMA status=41<READY,ERROR> error=4<ABORTED> LBA=20159

ad1: FAILURE - WRITE_DMA status=41<READY,ERROR> error=4<ABORTED> LBA=20191

ad1: FAILURE - WRITE_DMA status=41<READY,ERROR> error=4<ABORTED> LBA=20447

ad1: FAILURE - WRITE_DMA status=41<READY,ERROR> error=4<ABORTED> LBA=20703

ad1: FAILURE - WRITE_DMA status=41<READY,ERROR> error=4<ABORTED> LBA=20735

ad1: FAILURE - WRITE_DMA status=41<READY,ERROR> error=4<ABORTED> LBA=20831

ad1: FAILURE - WRITE_DMA status=41<READY,ERROR> error=4<ABORTED> LBA=20863

ad1: FAILURE - WRITE_DMA status=41<READY,ERROR> error=4<ABORTED> LBA=20895

Solution

This syslog message is generated under the following circumstances:

  • A hardware fault occurs on the hard drive OR

  • Communication between the hard drive and the kernel or the Routing Engine (RE) and the main board fails.

In VC-Setup, DMA failure errors might also be seen when the cache flush or the cache is full. It might cause the device not to accept SSH credentials and Virtual-Chassis to go into the hang state.
PR for reference - https://prsearch.juniper.net/problemreport/PR1383608

1. If you have access to the device and you are able to run the commands on the console, please try the below to confirm issue with the hard disk or the software. Please ensure you have root access while using shell prompt. If it is a software issue, we can move to step 3.
KB13103 [juniper.net] -- [Junos] How to perform SMART test from the shell
https://supportportal.juniper.net/s/article/Junos-How-to-perform-SMART-test-from-the-shell


2. If the issue is related to hard-disk, try to do a power drain during a maintenance window and check if the issue clears. Usually this could be a transient hard-disk issue and the issue might clear post a reboot.


3. If the issue is related to software, try to do a recovery-install / format-install and check if that helps resolve the issue. (similar logs might also be noticed post a failed upgrade when the hard-disk fails to load)
KB36947 [juniper.net] - [QFX] QFX5100/QFX5200 Junos recovery
https://supportportal.juniper.net/s/article/QFX-QFX5100-QFX5200-Junos-recovery?language=en_US

KB34749 [juniper.net] - [EX/QFX] Procedure to format install QFX5K device using a USB
https://supportportal.juniper.net/s/article/EX-QFX-Procedure-to-format-install-QFX5K-device-using-a-USB?language=en_US


4. If the issue still persists and the logs are continuously seen, or it re-occurs post recovery or power drain of the device (multiple occurrences), then we must proceed with the RMA as it could be a permanent hardware issue. Contact JTAC for further assistance

Modification History

2023-06-27 : Added relevant details (such as meaning of the log message and when it is reported, related PRs etc.) and procedures that can be used to confirm if this is a hard-disk fault or a SW issue.