Description

This article gives the reason behind why you observe the log message "kernel: end_request: I/O error, dev sdb, sector" along with "kernel: ata6: COMRESET failed (errno=-16)", and provides workaround for the same.

Symptoms

1) Kernel Logs shows both the said log messages "kernel: ata6: COMRESET failed (errno=-16)" and "kernel: end_request: I/O error, dev sdb, sector" as below:

2024-09-11T03:01:49.564651+00:00 ilulds103-node kernel: ata6: hard resetting link

2024-09-11T03:01:54+00:00 dcpfe0 kernel: ata6: link is slow to respond, please be patient (ready=0)

2024-09-11T03:01:59.646452+00:00 ilulds103-node kernel: ata6: hard resetting link

2024-09-11T03:01:59+00:00 dcpfe0 kernel: ata6: COMRESET failed (errno=-16)

2024-09-11T03:02:05.001074+00:00 ilulds103-node kernel: ata6: link is slow to respond, please be patient (ready=0)

2024-09-11T03:02:09.728407+00:00 ilulds103-node kernel: ata6: hard resetting link

2024-09-11T03:02:09+00:00 dcpfe0 kernel: ata6: COMRESET failed (errno=-16)

2024-09-11T03:02:15.083072+00:00 ilulds103-node kernel: ata6: link is slow to respond, please be patient (ready=0)

2024-09-11T03:02:44.801469+00:00 ilulds103-node kernel: ata6: COMRESET failed (errno=-16)

2024-09-11T03:02:44.801492+00:00 ilulds103-node kernel: ata6: limiting SATA link speed to 1.5 Gbps

2024-09-11T03:02:44.801496+00:00 ilulds103-node kernel: ata6: hard resetting link

2024-09-11T03:02:49+00:00 dcpfe0 kernel: ata6: COMRESET failed (errno=-16)

2024-09-11T03:02:49+00:00 dcpfe0 kernel: ata6: reset failed, giving up

 

 2024-09-11T03:02:49+00:00 dcpfe0 kernel: end_request: I/O error, dev sdb, sector 29782088

2024-09-11T03:02:49+00:00 dcpfe0 kernel: Aborting journal on device dm-0-8.

2024-09-11T03:02:49+00:00 dcpfe0 kernel: EXT4-fs error (device dm-0) in ext4_reserve_inode_write:4915: Journal has aborted

 

2) Check the host logs to find if the DMA Script Logs are present under 'disk_monitor.log' file as shown below:
 

2024-09-11 03:03:03 Failed to create a file in /junos directory. System will go for reboot after threshold_touch attempts

2024-09-11 03:03:08 Failed to create a file in /junos directory. System will go for reboot after threshold_touch attempts

2024-09-11 03:03:13 Failed to create a file in /junos directory. System will go for reboot after threshold_touch attempts

2024-09-11 03:03:13 Failed to create a file in i/junos directory after threshold_touch attempts. Initiating reboot...

2024-09-11 03:03:13 Reboot initiated due to disk sdb failure <<<<<<<<<<<

2024-09-11 03:03:19 Failed to create a file in /junos directory. System will go for reboot after threshold_touch attempts

2024-09-11 03:03:19 Reboot initiated due to disk sdb failure


If you look at the time stamps, Kernel records I/O error at 03:02 post which the DMA Write Script has been triggered and have initialized automatic reboot at 03:03.
 

Solution

In the event of above said symptoms which is due to the I/O error in the sdb (Write DMA error) , the DMA script has been triggered to do a automatic reboot but the box failed to come up.

In this scenario, the only way to recover is to power cycle the device.

Modification History

2024-09-23 : Article Created