We saw a Kernel error in LogWatch and wanted to investigate closely. This is dedicated CentOS 6.5 machine. The error we got was

WARNING: Kernel Errors Present
res 51/40:18:68:f2:8d/00:00:00:00:00/40 Emask 0x409 (media error)
<F> ...: 1 Time(s)
res 51/40:40:c0:28:8e/00:00:00:00:00/40 Emask 0x409 (media error)
<F> ...: 6 Time(s)
res 51/40:48:38:34:8e/00:00:00:00:00/40 Emask 0x409 (media error)
<F> ...: 5 Time(s)
ata2.00: error: { UNC } ...: 1 Time(s)
ata2: SError: { 10B8B } ...: 1 Time(s)
ata3.00: error: { UNC } ...: 11 Time(s)
sd 2:0:0:0: [sdb] Add. Sense: Unrecovered read error - auto reallocat ...:
2 Time(s)
sd 2:0:0:0: [sdb] Sense Key : Medium Error [current] [descr ...: 2 Time(s)

We wanted to check the hard drives so we used smartctl to check the disks. smartctl controls the Self-Monitoring, Analysis and Reporting Technology (SMART) system built into many newer hard drives. The machine is under RAID 1, so we checked each drive, one at a time. First we checked the first disk by running a long smartctl test.

> smartctl -t long /dev/sg0

After ~2 hours, we reviewed the output by printing all SMART information about the disk.

> smartctl -a /dev/sg0

This command outputs a lot of information. Here’s a snippet of our results, which says the long test results are good, and we have nothing to worry about.

<snip...>
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 2924 -
<snip...>

We then repeated the process for the second disk.

> smartctl -t long /dev/sg1
> smartctl -a /dev/sg1

The output showed a read failure, and a 90% remaining life. This means we need to look at replacing the second disk.

<snip...>
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%      2827         9316552
<snip...>