In a RAID array I had a have periodically lost a drive here and there over the past several months.  I was always able to readd and resync without losing data.  However at some point it looks like some minor corruption happened and this makes DRBD unhappy.

Using fsck did not help either.


Dec 19 06:01:45 storageboxtest4 kernel: [19005.945890] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22184379, block=88735771
Dec 19 06:01:45 storageboxtest4 kernel: [19005.946084] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22184380, block=88735771
Dec 19 06:01:45 storageboxtest4 kernel: [19005.946289] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22184381, block=88735771
Dec 19 06:01:46 storageboxtest4 kernel: [19006.903899] block drbd0: helper command: /sbin/drbdadm pri-on-incon-degr minor-0 exit code 0 (0x0)
Dec 19 06:01:46 storageboxtest4 kernel: [19006.903918] block drbd0: helper command: /sbin/drbdadm local-io-error minor-0
Dec 19 06:01:46 storageboxtest4 kernel: [19006.908202] Power Off
Dec 19 06:01:46 storageboxtest4 kernel: [19006.910016] md: md125: recovery interrupted.
Dec 19 06:01:47 storageboxtest4 kernel: [19007.715470] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22840092, block=91357233
Dec 19 06:01:47 storageboxtest4 kernel: [19007.715789] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22840092, block=91357233
Dec 19 06:01:47 storageboxtest4 kernel: [19007.785375] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22840094, block=91357233
Dec 19 06:01:47 storageboxtest4 kernel: [19007.785705] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22840094, block=91357233
Dec 19 13:35:55 storageboxtest4 kernel: imklog 5.8.10, log source = /proc/kmsg started.

Just a simple dd test gives back an io error near the end of the array:

dd: reading `/dev/drbd0': Input/output error

drbd, errors, corruptionin, raid, array, periodically, readd, resync, corruption, fsck, dec, storageboxtest, kernel, ext, fs, _get_inode_loc, unable, inode, helper, sbin, drbdadm, pri, incon, degr, io, md, interrupted, imklog, proc, kmsg, dd, dev, input, output,

Latest Articles