Description

Occasionally, QFX5K crashes (goes in an indeterministic state), and the following messages are seen on the console.

end_request: I/O error, dev sda, sector 9456388
end_request: I/O error, dev sda, sector 8039468
end_request: I/O error, dev sda, sector 0
end_request: I/O error, dev sda, sector 0
end_request: I/O error, dev sda, sector 0
end_request: I/O error, dev sda, sector 9456388
end_request: I/O error, dev sda, sector 8039468
end_request: I/O error, dev sda, sector 0
WRITE_DMA. ACB: ca 00 58 6d 20 40 00 00 00 00 10 00
CAM status: ATA Status Error
ATA status: 41 (DRDY ERR), error: 04 (ABRT )



 
Symptoms
These are transient hardware errors encountered by SSDs/ Hard drives. The common symptoms are:



  • Not able to SSH (login prompt not seen)
  • Not able to log in via console
  • Login prompt not seen on console
  • Can log in via console but FPC is missing
Solution
If the device is stuck in this mode then perform a power cycle to recover. If the device does not recover after a power cycle, then RMA is the only solution. 


It is generally safe to keep the chassis in service if power cycle helps in recovery but if the same issue is encountered more than once, it is recommended to replace the chassis. 


However if multiple failures are seen with the same partition or alternate we will want to engage escalation to understand if there is a batch problem.
Modification History
21/9 - Added power cycle as the recovery step.


6/28 - added a couple more console logs seen during this issue


 


6/30 - added content to solution