DRBD Errors Caused By Physical Corruption

In a RAID array I had a have periodically lost a drive here and there over the past several months.  I was always able to readd and resync without losing data.  However at some point it looks like some minor corruption happened and this makes DRBD unhappy.

Using fsck did not help either.

 


Dec 19 06:01:45 storageboxtest4 kernel: [19005.945890] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22184379, block=88735771
Dec 19 06:01:45 storageboxtest4 kernel: [19005.946084] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22184380, block=88735771
Dec 19 06:01:45 storageboxtest4 kernel: [19005.946289] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22184381, block=88735771
Dec 19 06:01:46 storageboxtest4 kernel: [19006.903899] block drbd0: helper command: /sbin/drbdadm pri-on-incon-degr minor-0 exit code 0 (0x0)
Dec 19 06:01:46 storageboxtest4 kernel: [19006.903918] block drbd0: helper command: /sbin/drbdadm local-io-error minor-0
Dec 19 06:01:46 storageboxtest4 kernel: [19006.908202] Power Off
Dec 19 06:01:46 storageboxtest4 kernel: [19006.910016] md: md125: recovery interrupted.
Dec 19 06:01:47 storageboxtest4 kernel: [19007.715470] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22840092, block=91357233
Dec 19 06:01:47 storageboxtest4 kernel: [19007.715789] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22840092, block=91357233
Dec 19 06:01:47 storageboxtest4 kernel: [19007.785375] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22840094, block=91357233
Dec 19 06:01:47 storageboxtest4 kernel: [19007.785705] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22840094, block=91357233
Dec 19 13:35:55 storageboxtest4 kernel: imklog 5.8.10, log source = /proc/kmsg started.

Just a simple dd test gives back an io error near the end of the array:

dd: reading `/dev/drbd0': Input/output error


Tags:

drbd, errors, corruptionin, raid, array, periodically, readd, resync, corruption, fsck, dec, storageboxtest, kernel, ext, fs, _get_inode_loc, unable, inode, helper, sbin, drbdadm, pri, incon, degr, io, md, interrupted, imklog, proc, kmsg, dd, dev, input, output,

Latest Articles

  • Debian Ubuntu Mint Howto Create Bridge (br0)
  • How To Control Interface that dhcpd server listens to on Debian based Linux like Mint and Ubuntu
  • LUKS unable to type password to unlock during boot on Debian, Ubuntu and Mint
  • Debian Ubuntu and Linux Mint Broken Kernel After Date - New Extra Module Naming Convention
  • Wordpress overwrites and wipes out custom htaccess rules and changes soluton
  • Apache htaccess and mod_rewrite how to redirect and force all URLs and visitors to the SSL / HTTPS version
  • python 3 pip cannot install mysql module
  • QEMU-KVM won't boot Windows 2016 or 2019 server on an Intel Core i3
  • Virtualbox vbox not starting "No suitable module for running kernel found"
  • Bind / named not responding to queries solution
  • Linux Mint How To Set Desktop Background Image From Bash Prompt CLI
  • ImageMagick Convert PDF Not Authorized
  • ImageMagick Converted PDF to JPEG some files have a black background solution
  • Linux Mint Mate Customize the Lock screen messages and hide username and real name
  • Ubuntu/Gnome/Mint/Centos How To Take a partial screenshot
  • ssh how to verify your host key / avoid MIM attacks
  • Cisco IP Phone CP-8845 8800/8900 Series How To Reset To Factory Settings Instructions
  • ls how to list ONLY directories
  • How to encrypt your SSH private key file id_rsa
  • Linux Mint 18 Disable User Name List from showing on Login Screen