DRBD Errors Caused By Physical Corruption

In a RAID array I had a have periodically lost a drive here and there over the past several months.  I was always able to readd and resync without losing data.  However at some point it looks like some minor corruption happened and this makes DRBD unhappy.

Using fsck did not help either.

 


Dec 19 06:01:45 storageboxtest4 kernel: [19005.945890] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22184379, block=88735771
Dec 19 06:01:45 storageboxtest4 kernel: [19005.946084] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22184380, block=88735771
Dec 19 06:01:45 storageboxtest4 kernel: [19005.946289] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22184381, block=88735771
Dec 19 06:01:46 storageboxtest4 kernel: [19006.903899] block drbd0: helper command: /sbin/drbdadm pri-on-incon-degr minor-0 exit code 0 (0x0)
Dec 19 06:01:46 storageboxtest4 kernel: [19006.903918] block drbd0: helper command: /sbin/drbdadm local-io-error minor-0
Dec 19 06:01:46 storageboxtest4 kernel: [19006.908202] Power Off
Dec 19 06:01:46 storageboxtest4 kernel: [19006.910016] md: md125: recovery interrupted.
Dec 19 06:01:47 storageboxtest4 kernel: [19007.715470] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22840092, block=91357233
Dec 19 06:01:47 storageboxtest4 kernel: [19007.715789] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22840092, block=91357233
Dec 19 06:01:47 storageboxtest4 kernel: [19007.785375] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22840094, block=91357233
Dec 19 06:01:47 storageboxtest4 kernel: [19007.785705] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22840094, block=91357233
Dec 19 13:35:55 storageboxtest4 kernel: imklog 5.8.10, log source = /proc/kmsg started.

Just a simple dd test gives back an io error near the end of the array:

dd: reading `/dev/drbd0': Input/output error


Tags:

drbd, errors, corruptionin, raid, array, periodically, readd, resync, corruption, fsck, dec, storageboxtest, kernel, ext, fs, _get_inode_loc, unable, inode, helper, sbin, drbdadm, pri, incon, degr, io, md, interrupted, imklog, proc, kmsg, dd, dev, input, output,

Latest Articles

  • Linux Mint Mate Customize the Lock screen messages and hide username and real name
  • Ubuntu/Gnome/Mint/Centos How To Take a partial screenshot
  • ssh how to verify your host key / avoid MIM attacks
  • Cisco IP Phone CP-8845 8800/8900 Series How To Reset To Factory Settings Instructions
  • ls how to list ONLY directories
  • How to encrypt your SSH private key file id_rsa
  • Linux Mint 18 Disable User Name List from showing on Login Screen
  • Firefox Cannot Hit Enter Key In Address Bar and Location History Not Working
  • Cisco Unified Communications Manager / CUCM IP 8.6,10,12 Install Error Solution
  • Ubuntu Debian Mint Linux SSHD OpenSSH Server Not Starting After Reboot Solution
  • nmap how to scan for all ports and not just the 1000 most common ports
  • Windows 7,8,10 and Server 2008, 2012, 2016, 2019 Read Only Attribute Won't Go Away
  • bind / named how to make a wildcard record and retain defined A records
  • Cisco Unified Communications Manager 12 Install Errors on Proxmox/KVM
  • Local Vs Universally Administered MAC Address NIC Refuses to come up
  • Cisco Unified Communications Manager 12 CUCM 12 - How To Enable Video Calling
  • Windows 7, 8, 10, Windows Server 2008, 2012, 2016, 2019 How To AC97 Audio Drivers and Other Unsigned Drivers
  • Cisco Unified Communications Manager / CUCM IP Telephony Definitions
  • tftp Linux xinetd verbose logging
  • Linux delete unused tap devices automatically