DRBD Errors Caused By Physical Corruption

In a RAID array I had a have periodically lost a drive here and there over the past several months.  I was always able to readd and resync without losing data.  However at some point it looks like some minor corruption happened and this makes DRBD unhappy.

Using fsck did not help either.

 


Dec 19 06:01:45 storageboxtest4 kernel: [19005.945890] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22184379, block=88735771
Dec 19 06:01:45 storageboxtest4 kernel: [19005.946084] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22184380, block=88735771
Dec 19 06:01:45 storageboxtest4 kernel: [19005.946289] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22184381, block=88735771
Dec 19 06:01:46 storageboxtest4 kernel: [19006.903899] block drbd0: helper command: /sbin/drbdadm pri-on-incon-degr minor-0 exit code 0 (0x0)
Dec 19 06:01:46 storageboxtest4 kernel: [19006.903918] block drbd0: helper command: /sbin/drbdadm local-io-error minor-0
Dec 19 06:01:46 storageboxtest4 kernel: [19006.908202] Power Off
Dec 19 06:01:46 storageboxtest4 kernel: [19006.910016] md: md125: recovery interrupted.
Dec 19 06:01:47 storageboxtest4 kernel: [19007.715470] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22840092, block=91357233
Dec 19 06:01:47 storageboxtest4 kernel: [19007.715789] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22840092, block=91357233
Dec 19 06:01:47 storageboxtest4 kernel: [19007.785375] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22840094, block=91357233
Dec 19 06:01:47 storageboxtest4 kernel: [19007.785705] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22840094, block=91357233
Dec 19 13:35:55 storageboxtest4 kernel: imklog 5.8.10, log source = /proc/kmsg started.

Just a simple dd test gives back an io error near the end of the array:

dd: reading `/dev/drbd0': Input/output error


Tags:

drbd, errors, corruptionin, raid, array, periodically, readd, resync, corruption, fsck, dec, storageboxtest, kernel, ext, fs, _get_inode_loc, unable, inode, helper, sbin, drbdadm, pri, incon, degr, io, md, interrupted, imklog, proc, kmsg, dd, dev, input, output,

Latest Articles

  • Linux Ubuntu Cannot Print Large Images
  • Cannot Print PDF Solution and Howto Resize
  • Linux Console Login Screen TTY Change Message
  • Apache Cannot Start Listening Already on 0.0.0.0
  • MySQL Bash Query to pipe input directly without using heredoc trick
  • CentOS 6 and 7 / RHEL Persistent DHCP Solution
  • Debian Ubuntu Mint rc-local service startup error solution rc-local.service: Failed at step EXEC spawning /etc/rc.local: Exec format error
  • MySQL Cheatsheet Guide and Tutorial
  • bash script kill whois or other command that is running for too long
  • Linux tftp listens on all interfaces and IPs by DEFAULT Security Risk Hole Solution
  • python import docx error
  • Cisco Unified Communications Manager Express Cheatsheet CUCME CME
  • Linux Ubuntu Debian Missing privilege separation directory: /var/run/sshd
  • bash how to count the number of columns or words in a line
  • bash if statement how to test program output without assigning to variable
  • RTNETLINK answers: Network is unreachable
  • Centos 7 how to save iptables rules like Centos 6
  • nfs tuning maximum amount of connections
  • qemu-kvm error "Could not initialize SDL(No available video device) - exiting"
  • Centos 7 tftpd will not work with selinux enabled