This happened during a RAID array check:
SMART says both drives pass the test, but I'm doing a long test on them and hopefully this is not a hardware error.
Apr 3 04:22:01 remote kernel: md: syncing RAID array md2
Apr 3 04:22:01 remote kernel: md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc.
Apr 3 04:22:01 remote kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction.
Apr 3 04:22:01 remote kernel: md: using 128k window, over a total of 488287552 blocks.
Apr 3 04:22:01 remote kernel: md: delaying resync of md1 until md2 has finished resync (they share one or more physical units)
Apr 3 04:29:28 remote ntpd[7131]: synchronized to 24.215.0.24, stratum 2
Apr 3 05:25:01 remote auditd[6570]: Audit daemon rotating log files
Apr 3 08:35:07 remote kernel: ata4.00: exception Emask 0x0 SAct 0x7fffffff SErr 0x0 action 0x0
Apr 3 08:35:07 remote kernel: ata4.00: irq_stat 0x40000008
Apr 3 08:35:07 remote kernel: ata4.00: cmd 60/80:80:c0:41:69/00:00:42:00:00/40 tag 16 ncq 65536 in
Apr 3 08:35:07 remote kernel: res 41/40:00:38:42:69/cd:00:42:00:00/40 Emask 0x409 (media error) <F>
Apr 3 08:35:07 remote kernel: ata4.00: status: { DRDY ERR }
Apr 3 08:35:07 remote kernel: ata4.00: error: { UNC }
Apr 3 08:35:07 remote kernel: ata4.00: configured for UDMA/133
Apr 3 08:35:07 remote kernel: ata4: EH complete
Apr 3 08:35:07 remote kernel: SCSI device sdb: 3907029168 512-byte hdwr sectors (2000399 MB)
Apr 3 08:35:07 remote kernel: sdb: Write Protect is off
Apr 3 08:35:07 remote kernel: SCSI device sdb: drive cache: write back
Apr 3 08:35:09 remote kernel: ata4.00: exception Emask 0x0 SAct 0x5f8079ff SErr 0x0 action 0x0
Apr 3 08:35:21 remote kernel: ata4.00: irq_stat 0x40000008
Apr 3 08:35:21 remote kernel: ata4.00: cmd 60/80:70:c0:41:69/00:00:42:00:00/40 tag 14 ncq 65536 in
Apr 3 08:35:21 remote kernel: res 41/40:00:38:42:69/cd:00:42:00:00/40 Emask 0x409 (media error) <F>
Apr 3 08:35:21 remote kernel: ata4.00: status: { DRDY ERR }
Apr 3 08:35:21 remote kernel: ata4.00: error: { UNC }
Apr 3 08:35:21 remote kernel: ata4.00: configured for UDMA/133
Apr 3 08:35:21 remote kernel: ata4: EH complete
Apr 3 08:35:21 remote kernel: SCSI device sdb: 3907029168 512-byte hdwr sectors (2000399 MB)
Apr 3 08:35:21 remote kernel: sdb: Write Protect is off
Apr 3 08:35:21 remote kernel: SCSI device sdb: drive cache: write back
Apr 3 08:35:21 remote kernel: ata4.00: exception Emask 0x0 SAct 0xfffff SErr 0x0 action 0x0
Apr 3 08:35:21 remote kernel: ata4.00: irq_stat 0x40000008
Apr 3 08:35:21 remote kernel: ata4.00: cmd 60/80:38:c0:41:69/00:00:42:00:00/40 tag 7 ncq 65536 in
Apr 3 08:35:21 remote kernel: res 41/40:00:38:42:69/cd:00:42:00:00/40 Emask 0x409 (media error) <F>
Apr 3 08:35:21 remote kernel: ata4.00: status: { DRDY ERR }
Apr 3 08:35:21 remote kernel: ata4.00: error: { UNC }
Apr 3 08:35:21 remote kernel: ata4.00: configured for UDMA/133
Apr 3 08:35:21 remote kernel: ata4: EH complete
Apr 3 08:35:21 remote kernel: SCSI device sdb: 3907029168 512-byte hdwr sectors (2000399 MB)
Apr 3 08:35:21 remote kernel: sdb: Write Protect is off
Apr 3 08:35:21 remote kernel: SCSI device sdb: drive cache: write back
Apr 3 08:35:21 remote kernel: ata4.00: exception Emask 0x0 SAct 0xb9800 SErr 0x0 action 0x0
Apr 3 08:35:21 remote kernel: ata4.00: irq_stat 0x40000008
Apr 3 08:35:21 remote kernel: ata4.00: cmd 60/80:60:c0:41:69/00:00:42:00:00/40 tag 12 ncq 65536 in
Apr 3 08:35:22 remote kernel: res 41/40:00:38:42:69/cd:00:42:00:00/40 Emask 0x409 (media error) <F>
Apr 3 08:35:22 remote kernel: ata4.00: status: { DRDY ERR }
Apr 3 08:35:22 remote kernel: ata4.00: error: { UNC }
Apr 3 08:35:22 remote kernel: ata4.00: configured for UDMA/133
Apr 3 08:35:22 remote kernel: ata4: EH complete
Apr 3 08:35:22 remote kernel: SCSI device sdb: 3907029168 512-byte hdwr sectors (2000399 MB)
Apr 3 08:35:22 remote kernel: sdb: Write Protect is off
Apr 3 08:35:22 remote kernel: SCSI device sdb: drive cache: write back
Apr 3 08:35:22 remote kernel: ata4.00: exception Emask 0x0 SAct 0x3f SErr 0x0 action 0x0
Apr 3 08:35:22 remote kernel: ata4.00: irq_stat 0x40000008
Apr 3 08:35:22 remote kernel: ata4.00: cmd 60/80:20:c0:41:69/00:00:42:00:00/40 tag 4 ncq 65536 in
Apr 3 08:35:22 remote kernel: res 41/40:00:38:42:69/cd:00:42:00:00/40 Emask 0x409 (media error) <F>
Apr 3 08:35:22 remote kernel: ata4.00: status: { DRDY ERR }
Apr 3 08:35:22 remote kernel: ata4.00: error: { UNC }
Apr 3 08:35:22 remote kernel: ata4.00: configured for UDMA/133
Apr 3 08:35:22 remote kernel: ata4: EH complete
Apr 3 08:35:22 remote kernel: SCSI device sdb: 3907029168 512-byte hdwr sectors (2000399 MB)
Apr 3 08:35:22 remote kernel: sdb: Write Protect is off
Apr 3 08:35:22 remote kernel: SCSI device sdb: drive cache: write back
Apr 3 08:35:22 remote kernel: ata4.00: exception Emask 0x0 SAct 0x32 SErr 0x0 action 0x0
Apr 3 08:35:22 remote kernel: ata4.00: irq_stat 0x40000008
Apr 3 08:35:22 remote kernel: ata4.00: cmd 60/80:08:c0:41:69/00:00:42:00:00/40 tag 1 ncq 65536 in
Apr 3 08:35:22 remote kernel: res 41/40:00:38:42:69/cd:00:42:00:00/40 Emask 0x409 (media error) <F>
Apr 3 08:35:22 remote kernel: ata4.00: status: { DRDY ERR }
Apr 3 08:35:22 remote kernel: ata4.00: error: { UNC }
Apr 3 08:35:22 remote kernel: ata4.00: configured for UDMA/133
Apr 3 08:35:22 remote kernel: sd 3:0:0:0: SCSI error: return code = 0x08000002
Apr 3 08:35:22 remote kernel: sdb: Current [descriptor]: sense key: Medium Error
Apr 3 08:35:22 remote kernel: Add. Sense: Unrecovered read error - auto reallocate failed
Apr 3 08:35:22 remote kernel:
Apr 3 08:35:22 remote kernel: Descriptor sense data with sense descriptors (in hex):
Apr 3 08:35:22 remote kernel: 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
Apr 3 08:35:22 remote kernel: 42 69 42 38
Apr 3 08:35:22 remote kernel: end_request: I/O error, dev sdb, sector 1114194488
Apr 3 08:35:22 remote kernel: ata4: EH complete
Apr 3 08:35:22 remote kernel: SCSI device sdb: 3907029168 512-byte hdwr sectors (2000399 MB)
Apr 3 08:35:22 remote kernel: sdb: Write Protect is off
Apr 3 08:35:22 remote kernel: SCSI device sdb: drive cache: write back
Apr 3 08:46:45 remote kernel: md: md2: sync done.
Apr 3 08:46:45 remote kernel: md: syncing RAID array md1
Apr 3 08:46:45 remote kernel: md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc.
Apr 3 08:46:45 remote kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction.
Apr 3 08:46:45 remote kernel: md: using 128k window, over a total of 29302464 blocks.
Apr 3 08:46:45 remote kernel: RAID1 conf printout:
Apr 3 08:46:45 remote kernel: --- wd:2 rd:2
Apr 3 08:46:45 remote kernel: disk 0, wo:0, o:1, dev:sda3
Apr 3 08:46:45 remote kernel: disk 1, wo:0, o:1, dev:sdb3
Apr 3 08:51:54 remote kernel: md: md1: sync done.
Apr 3 08:51:54 remote kernel: RAID1 conf printout:
Apr 3 08:51:54 remote kernel: --- wd:2 rd:2
Apr 3 08:51:54 remote kernel: disk 0, wo:0, o:1, dev:sda2
Apr 3 08:51:54 remote kernel: disk 1, wo:0, o:1, dev:sdb2
wdears, raid, array, hardware, apr, kernel, md, syncing, _guaranteed_, reconstruction, kb, disc, maximum, idle, io, bandwidth, delaying, resync, ntpd, synchronized, stratum, auditd, audit, daemon, rotating, ata, exception, emask, sact, fffffff, serr, irq_stat, cmd, ncq, res, drdy, err, unc, configured, udma, scsi, sdb, byte, hdwr, sectors, mb, cache, ff, xfffff, xb, descriptor, medium, unrecovered, auto, reallocate, descriptors, hex, end_request, dev, sector, sync, conf, printout, wd, rd, disk, wo, sda,