RE encounters WRITE

Description

RE encounters with WRITE_DMA error

Symptoms

These messages are logged when Junos tries to access the hardware memory (ada1) to write any data. The DMA stands for the direct memory access here. This log can indicate corruption, in which the device fails to read a certain part of the hard disk. This is a notification of command timeout and signals a hardware issue [transient or permanent].

Here is a snippet of the logs for reference:
Dec 13 20:25:32 MX480_re0 kernel: %KERN-2: (ada1:ata0:0:1:0): WRITE_DMA. ACB: ca 00 20 25 27 40 00 00 00 00 40 00
Dec 13 20:25:32 MX480_re0 kernel: %KERN-2: (ada1:ata0:0:1:0): CAM status: Command timeout
Dec 13 20:25:32 MX480_re0 kernel: %KERN-2: (ada1:ata0:0:1:0): Retrying command
Dec 13 20:31:14 MX480_re0 eventd: %SYSLOG-3: sendto: No route to host
Dec 13 20:31:14 MX480_re0 eventd[11767]: %SYSLOG-5-SYSTEM_ABNORMAL_SHUTDOWN: System abnormally shut down
Dec 13 20:31:14 MX480_re0 eventd[11767]: %SYSLOG-5-SYSTEM_OPERATIONAL: System is operational

Solution

1. Check to see if it's a transient or permanent hardware issue by verifying the disk integrity as follows:
Login to RE or RE Linux host shell
>start shell user root
%vhclient -s *
#smartctl -a /dev/sda
#smartctl -a /dev/sdb

*Command varies depending on the RE model.... For instance:
RE-QFX1000* use "hshell"
NG-RE use "vhclient -s"
Legacy RE - run cmd in RE shell as a root user

Tip: Run smartctl -h and show chassis hardware extensive for model-specific info

Example [Fron NG-RE]:
-sh-4.1# smartctl -a /dev/sda
smartctl 5.42 2011-10-20 r3458 [x86_64-linux-2.6.32-279.22.1.el6.x86_64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Device Model:   SATA SSD
Serial Number:  EB9807560AF000524852
Firmware Version: S9FM01.3
User Capacity:  16,013,942,784 bytes [16.0 GB]
Sector Size:   512 bytes logical/physical
Device is:    Not in smartctl database [for details use: -P showall]
ATA Version is:  8
ATA Standard is: ACS-3 (revision not indicated)
Local Time is:  Thu Apr 13 17:21:32 2023 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x02) Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:   (  0) The previous self-test routine completed
                    without error or no self-test has ever
                    been run. ---> 0 indicates that test is completed or currently no tests are running
Total time to complete Offline
data collection:        (  30) seconds.
Offline data collection
capabilities:          (0x1b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    No Selective Self-test supported.
SMART capabilities:      (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:    (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:    (  1) minutes.
Extended self-test routine
recommended polling time:    (  2) minutes.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME     FLAG   VALUE WORST THRESH TYPE   UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate   0x000a  100  100  000  Old_age  Always    -    0
9 Power_On_Hours     0x0012  100  100  000  Old_age  Always    -    34495
12 Power_Cycle_Count    0x0012  100  100  000  Old_age  Always    -    119
168 Unknown_Attribute    0x0012  100  100  000  Old_age  Always    -    0
170 Unknown_Attribute    0x0013  100  100  010  Pre-fail Always    -    10
173 Unknown_Attribute    0x0000  100  100  000  Old_age  Offline   -    48759202
192 Power-Off_Retract_Count 0x0012  100  100  000  Old_age  Always    -    10
218 Unknown_Attribute    0x0000  100  100  000  Old_age  Offline   -    0
241 Total_LBAs_Written   0x0012  100  100  000  Old_age  Always    -    1671010

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description  Status         Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline  Completed without error    00%    742     -
# 2 Short offline    Completed without error    00%    694     -
# 3 Short offline    Completed without error    00%    694     -
# 4 Extended offline  Completed without error    00%    692     -
# 5 Short offline    Completed without error    00%    692     -
# 6 Short offline    Aborted by host        00%    692     -
# 7 Short offline    Completed without error    00%    692     -
# 8 Short offline    Completed without error    00%    692     -
# 9 Short offline    Aborted by host        00%    692     -
#10 Short offline    Completed without error    00%    692     -
#11 Short offline    Completed without error    00%    692     -
#12 Short offline    Completed without error    00%    691     -
#13 Short offline    Completed without error    00%    691     -

## In NG-REs hard-disk tests can be executed from cli via
request vmhost hard-disk-test [long or short] disk /dev/sdb
request vmhost hard-disk-test show-status disk /dev/sdb
request vmhost hard-disk-test [long or short] disk /dev/sda
request vmhost hard-disk-test show-status disk /dev/sda

For instance:
labroot@bng-jtac-mx960-r2035-re0> request vmhost hard-disk-test short disk /dev/sdb
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-5.2.60-rt15-LTS19] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Short self-test routine immediately in off-line mode".
Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 1 minutes for test to complete.
Test will complete after Mon Aug 19 14:29:36 2024

labroot@bng-jtac-mx960-r2035-re0> request vmhost hard-disk-test show-status disk /dev/sdb
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-5.2.60-rt15-LTS19] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: SwissBit Slim Sata SSD
Device Model: SFSA200GV3AA4TO-I-HC-246-JUN
Serial Number: 000060220869A4000047
Firmware Version: SBR12050
User Capacity: 200,049,647,616 bytes [200 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Aug 19 14:32:46 2024 IST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x02) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 249) Self-test routine in progress...
90% of test remaining. ----> Indicates that only 10% of self-test is completed
Total time to complete Offline
data collection: ( 57) seconds.
Offline data collection
capabilities: (0x71) SMART execute Offline immediate.
No Auto Offline data collection support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0002) Does not save SMART data before
entering power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 1) minutes.
Conveyance self-test routine
recommended polling time: ( 1) minutes.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 000 Pre-fail Always - 0
5 Reallocated_Sector_Count0x0013 100 100 000 Pre-fail Always - 0
9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 5923
12 Power_Cycle_Count 0x0012 100 100 000 Old_age Always - 73
160 Uncorrectable_Sector_Count0x0002 100 100 000 Old_age Always - 0
161 Spare_Blocks 0x0012 100 100 025 Old_age Always - 569
163 Initial_Invalid_Blocks 0x0003 100 100 000 Pre-fail Always - 17
164 Total_Erase_Count 0x0000 100 100 000 Old_age Offline - 26209
165 Maxmimum_Erase_Count 0x0000 100 100 000 Old_age Offline - 20
166 Minimum_Erase_Count 0x0000 100 100 000 Old_age Offline - 0
167 Average_Erase_Count 0x0012 100 100 001 Old_age Always - 12
168 MaxSpecified_Erase_count0x0012 100 100 000 Old_age Always - 3000
169 PowerOn_UECC_Count 0x0003 100 100 000 Pre-fail Always - 35
192 Initial_Spare_Blk_Count 0x0000 100 100 000 Old_age Offline - 569
193 Dynamic_Remaps 0x0012 100 100 000 Old_age Always - 10
194 Temperature_Celsius 0x0023 100 100 000 Pre-fail Always - 4325148
195 Hardware_ECC_Recovered 0x0012 100 100 000 Old_age Always - 30643
196 Reallocated_Event_Count 0x0000 000 000 000 Old_age Offline - 0
198 Uncorrectable_Sect_Count0x0012 100 100 000 Old_age Always - 0
199 SATA_Phy_CRC_Error_Count0x0012 100 100 000 Old_age Always - 0
215 Trim_Count 0x0012 100 100 000 Old_age Always - 1884
235 TotalFlashLBA_Written 0x0012 100 100 000 Old_age Always - 47152288
237 TotalFlashLBA_Written_E 0x0012 100 100 000 Old_age Always - 0
241 TotalHostLBA_Write 0x0012 100 100 000 Old_age Always - 2968001902
242 TotalHostLBA_Read 0x0012 100 100 000 Old_age Always - 3251249045
243 TotalHostLBA_Written_E 0x0012 100 100 000 Old_age Always - 0
244 TotalHostLBA_Read_E 0x0012 100 100 000 Old_age Always - 0
248 SSD_Life_Left 0x0012 100 100 000 Old_age Always - 100
249 Spare_Block_Count 0x0013 100 100 000 Pre-fail Always - 100

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 35 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
7 0 65535 Read_scanning was completed without error
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

2. If no errors are seen, reboot the affected RE vmhost via request vmhost reboot re1
3. If the errors persist, perform a format install
4. If the issue persists even after step 3, proceed with the hard drive or RE replacement, depending on the model
5. Contact JTAC for assistance with issue isolation

Modification History

2023-12-22 : Article Created