This article provides the required steps to check the integration of HDD and SCSI controllers on QFabric, and also the steps to identify any problems on the hard drives in case of SCSI errors.
The symptoms could be any unprecedented behavior on QFabric. A few observations include the following:
HDD read write failure
VM malfunctioning (for example, NNG VM ungraceful switchover causing external l3 protocol to flap)
Node disconnection (again can be due to FM VM malfunctioning)
DG mastership switchover in an unregulated condition
Due to SCSI errors and the above symptoms, the network being down is highly possible.
Note: You need to log in to both DGs via the console or SSH. The checks on the DGs will not impact existing operations.
[root@dg0 /]# cd vmm [root@dg0 logs]# pwd /vmm/logs
[root@dg0 logs]# cat ccif_server.log | grep "Fri, May 11 2018" ( Collect the complete log. )
[root@dg0 logs]# cat ccif_server.log | grep "Fri, May 11 2018"
[root@dg0 /]# cat /var/log/messages | grep scsi ( Look for SCSI controller related errors. )
[root@dg0 /]# cat /var/log/messages | grep scsi
[root@dg0 /]# /pbdata/usr/mysql-5.5.28/data dg0.err dg1.err [labroot@dg0 ~]# cat /sys/block/sd*/device/timeout [labroot@dg0 ~]# cat /sys/block/sd*/device/timeout
Collect the SCSI commands data from the DG-0 and DG-1 shell level. The commands along with the expected outputs are as follows:
Note: In working condition, the volume must show as "OPTIMAL." It can show "DEGRADED" if one of the HDDs in either DG is not online.
[root@dg0 ~]# disk_status.sh -r vol_id:0B37315D2C885E21 OPTIMAL phys_id:0 ONLINE phys_id:1 ONLINE scsi_id:0 0% scsi_id:1 0% [root@dg0 ~]# /root/mpt-status -i 4 -s -n -p Checking for SCSI ID:0 Checking for SCSI ID:1 Checking for SCSI ID:2 Checking for SCSI ID:3 Checking for SCSI ID:4 Checking for SCSI ID:5 Checking for SCSI ID:6 Checking for SCSI ID:7 Checking for SCSI ID:8 Checking for SCSI ID:9 Checking for SCSI ID:10 vol_id:10 OPTIMAL phys_id:0 ONLINE phys_id:1 ONLINE scsi_id:0 100% scsi_id:1 100%
A few important utility commands (working state output) are as follows. Make sure that you capture the output on both DGs.
[root@dg1 ~]# lsutil -bash: lsutil: command not found [root@dg1 ~]# /root/mpt-status -i 4 -s -n -p Checking for SCSI ID:0 Checking for SCSI ID:1 Checking for SCSI ID:2 Checking for SCSI ID:3 Checking for SCSI ID:4 Checking for SCSI ID:5 Checking for SCSI ID:6 Checking for SCSI ID:7 Checking for SCSI ID:8 vol_id:8 OPTIMAL phys_id:1 ONLINE phys_id:0 ONLINE scsi_id:1 100% scsi_id:0 100%
[root@dg0 ~]# smartctl -a -T permissive /dev/sg0 <<< SG0 & SG1 define HDD0 & HDD1, respectively. Change attributes accordingly to check both HDDs.
smartctl version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: WDC WD2000FYYZ-01UL1B2 Serial Number: WD-WMC1P0E37UTL Firmware Version: 01.01K03 User Capacity: 2,000,398,934,016 bytes Device is: Not in smartctl database [for details use: -P show all] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Fri May 11 11:31:51 2018 UTC SMART support is: Available - device has SMART capability. SMART support is: Enabled Error SMART Status command failed Please get assistance from http://smartmontools.sourceforge.net/ Values from ATA Return Descriptor are: 00 09 0c 00 00 00 00 00 00 00 00 00 00 00 00 === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x84) Offline data collection activity was suspended by an interrupting command from host. Auto Offline Data Collection:Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (24660) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x70bd) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 169 166 021 Pre-fail Always - 6525 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 167 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 076 076 000 Old_age Always - 17710 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 167 183 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 165 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 1 194 Temperature_Celsius 0x0022 120 110 000 Old_age Always - 30 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
[root@dg0 ~]# lsiutil ( Follow the bold options to run the utility. )
[root@dg0 ~]# lsiutil
LSI Logic MPT Configuration Utility, Version 1.67, September 15, 2011 1 MPT Port found Port Name Chip Vendor/Type/Rev MPT Rev Firmware Rev IOC 1. /proc/mpt/ioc0 LSI Logic SAS1068E B3 105 01210000 0 Select a device: [1-1 or 0 to quit] 1 1. Identify firmware, BIOS, and/or FCode 2. Download firmware (update the FLASH) 4. Download/erase BIOS and/or FCode (update the FLASH) 8. Scan for devices 10. Change IOC settings (interrupt coalescing) 13. Change SAS IO Unit settings 16. Display attached devices 20. Diagnostics 21. RAID actions 22. Reset bus SAS1068E's links are 3.0 G, 3.0 G, down, down, down, down, down, down B___T___L Type Vendor Product Rev SASAddress PhyNum 0 10 0 Disk LSILOGIC Logical Volume 3000 Hidden RAID Devices: B___T Device Vendor Product Rev SASAddress PhyNum 0 11 PhysDisk 0 ATA WDC WD2000FYYZ-0 1K03 a2984920c1abc18c 1 0 12 PhysDisk 1 ATA WDC WD2000FYYZ-0 1K03 a2984921a6cccd88 0 Main menu, select an option: [1-99 or e/p/w or 0 to quit] 1 Current active firmware version is 011e0000 (1.30.00) Firmware image's version is MPTFW-01.30.00.00-IE LSI Logic x86 BIOS image's version is MPTBIOS-6.30.00.00 (2009.11.12) Main menu, select an option: [1-99 or e/p/w or 0 to quit] 0 Main menu, select an option: [1-99 or e/p/w or 0 to quit] 0 Port Name Chip Vendor/Type/Rev MPT Rev Firmware Rev IOC 1. /proc/mpt/ioc0 LSI Logic SAS1068E B3 105 011e0000 0 Select a device: [1-1 or 0 to quit] 1 Main menu, select an option: [1-99 or e/p/w or 0 to quit] 1. Identify firmware, BIOS, and/or FCode 2. Download firmware (update the FLASH) 4. Download/erase BIOS and/or FCode (update the FLASH) 8. Scan for devices 10. Change IOC settings (interrupt coalescing) 13. Change SAS IO Unit settings 16. Display attached devices 20. Diagnostics 21. RAID actions 22. Reset bus Port Name Chip Vendor/Type/Rev MPT Rev Firmware Rev IOC 1. /proc/mpt/ioc0 LSI Logic SAS1068E B3 105 011e0000 0 Main menu, select an option: [1-99 or e/p/w or 0 to quit] 16 SAS1068E's links are 3.0 G, 3.0 G, down, down, down, down, down, down B___T SASAddress PhyNum Handle Parent Type 500e08101037b7a0 0001 SAS Initiator 500e08101037b7a1 0002 SAS Initiator 500e08101037b7a2 0003 SAS Initiator 500e08101037b7a3 0004 SAS Initiator 500e08101037b7a4 0005 SAS Initiator 500e08101037b7a5 0006 SAS Initiator 500e08101037b7a6 0007 SAS Initiator 500e08101037b7a7 0008 SAS Initiator 0 12 a2984921a6cccd88 0 0009 0001 SATA Target 0 11 a2984920c1abc18c 1 000a 0002 SATA Target Type NumPhys PhyNum Handle PhyNum Handle Port Speed [Disk speed must match for both HDDs.] Adapter 8 0 0001 --> 0 0009 0 3.0 1 0002 --> 0 000a 1 3.0 [If there is speed mismatch, use KB32952 [juniper.net].] Enclosure Handle Slots SASAddress B___T (SEP) 0001 8 500e08101037b7a0 Main menu, select an option: [1-99 or e/p/w or 0 to quit] 0 Port Name Chip Vendor/Type/Rev MPT Rev Firmware Rev IOC 1. / proc/mpt/ioc0 LSI Logic SAS1068E B3 105 01210000 0 Select a device: [1-1 or 0 to quit] 1 1. Identify firmware, BIOS, and/or FCode 2. Download firmware (update the FLASH) 4. Download/erase BIOS and/or FCode (update the FLASH) 8. Scan for devices 10. Change IOC settings (interrupt coalescing) 13. Change SAS IO Unit settings 16. Display attached devices 20. Diagnostics 21. RAID actions Main menu, select an option: [1-99 or e/p/w or 0 to quit] 21 1. Show volumes 2. Show physical disks 1 volume is active, 2 physical disks are active Volume 0 is Bus 0 Target 10, Type IM (Integrated Mirroring) Volume Name: Volume WWID: 0b37315d2c885e21 Volume State: optimal, enabled Volume Settings: write caching disabled, auto configure Volume draws from Hot Spare Pools: 0 Volume Size 1907348 MB, 2 Members Primary is PhysDisk 0 (Bus 0 Target 11) Secondary is PhysDisk 1 (Bus 0 Target 12) RAID actions menu, select an option: [1-99 or e/p/w or 0 to quit] 2 1 volume is active, 2 physical disks are active PhysDisk 0 is Bus 0 Target 11 PhysDisk State: online PhysDisk Size 1907729 MB, Inquiry Data: ATA WDC WD2000FYYZ-0 1K03 Warning, page length in reply does not match page length in buffer (9 vs. 15) PhysDisk 1 is Bus 0 Target 12 PhysDisk State: online PhysDisk Size 1907729 MB, Inquiry Data: ATA WDC WD2000FYYZ-0 1K03 Warning, page length in reply does not match page length in buffer (9 vs. 15) RAID actions menu, select an option: [1-99 or e/p/w or 0 to quit] 3 Volume 0 is Bus 0 Target 10, Type IM (Integrated Mirroring) Volume 0 State: optimal, enabled