DRBD Split-brain solution

Uh oh

[17925926.174277] block drbd0: Handshake successful: Agreed network protocol version 96
[17925926.174325] block drbd0: conn( WFConnection -> WFReportParams )
[17925926.174342] block drbd0: Starting asender thread (from drbd0_receiver [1682])
[17925926.174432] block drbd0: data-integrity-alg: <not-used>
[17925926.174581] block drbd0: drbd_sync_handshake:
[17925926.174586] block drbd0: self 2AAE66AF9252D6DB:2815BF75CBEA0687:E303F24CF2826E13:E302F24CF2826E13 bits:73000 flags:0
[17925926.174591] block drbd0: peer 24F1983D580A1E83:2815BF75CBEA0686:E303F24CF2826E12:E302F24CF2826E13 bits:93874 flags:0
[17925926.174596] block drbd0: uuid_compare()=100 by rule 90
[17925926.174602] block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0
[17925926.180872] block drbd0: meta connection shut down by peer.
[17925926.181041] block drbd0: conn( WFReportParams -> NetworkFailure )
[17925926.181055] block drbd0: asender terminated
[17925926.181058] block drbd0: Terminating asender thread
[17925926.207698] block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0 exit code 0 (0x0)
[17925926.207704] block drbd0: Split-Brain detected but unresolved, dropping connection!
[17925926.207876] block drbd0: helper command: /sbin/drbdadm split-brain minor-0
[17925926.209653] block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)
[17925926.209659] block drbd0: conn( NetworkFailure -> Disconnecting )
[17925926.209668] block drbd0: error receiving ReportState, l: 4!
[17925926.209839] block drbd0: Connection closed
[17925926.209846] block drbd0: conn( Disconnecting -> StandAlone )
[17925926.209862] block drbd0: receiver terminated
[17925926.209865] block drbd0: Terminating receiver thread


[root@box18 ~]# cat /proc/drbd
version: 8.3.13 (api:88/proto:86-96)
GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by root@sighted, 2012-10-09 12:47:51
 0: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown A r-----
    ns:0 nr:0 dw:983961744 dr:1225316590 al:3645772 bm:110268 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:1619916
[root@box18 ~]# drbdadm connect r0
DRBD module version: 8.3.13
   userland version: 8.3.16
preferably kernel and userland versions should match.
0: Failure: (125) Device has a net-config (use disconnect first)
Command 'drbdsetup 0 net ipv4:192.168.5.2.18:7788 ipv4:192.168.5.2.17:7788 A --set-defaults --create-device --after-sb-0pri=discard-least-changes --after-sb-1pri=consensus' terminated with exit code 10


[root@box18 ~]# drbdadm connect r0
DRBD module version: 8.3.13
   userland version: 8.3.16
preferably kernel and userland versions should match.
0: Failure: (125) Device has a net-config (use disconnect first)
Command 'drbdsetup 0 net ipv4:192.168.5.2.18:7788 ipv4:192.168.5.2.17:7788 A --set-defaults --create-device --after-sb-0pri=discard-least-changes --after-sb-1pri=consensus' terminated with exit code 10
 

Solution


drbdadm disconnect r0

drbdadm -- --discard-my-data connect r0



#On the main active node


drbdadm connect r0
 cat /proc/drbd
version: 8.3.13 (api:88/proto:86-96)
GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by root@sighted, 2012-10-09 12:47:51
 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent A r-----
    ns:3580 nr:0 dw:1597648248 dr:12347577 al:12090 bm:58295 lo:13 pe:169 ua:85 ap:1 ep:1 wo:b oos:3289420
    [>....................] sync'ed:  0.2% (3289420/3290212)K
    finish: 1:08:31 speed: 792 (792) K/sec


#on the dead node that is rejoining


 cat /proc/drbd
version: 8.3.13 (api:88/proto:86-96)
GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by root@sighted, 2012-10-09 12:47:51
 0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate A r-----
    ns:0 nr:151088 dw:984106508 dr:1225316590 al:3645772 bm:110390 lo:61 pe:0 ua:2 ap:0 ep:3 wo:b oos:3170888
    [>....................] sync'ed:  3.8% (3170888/3290212)K
    finish: 0:11:48 speed: 4,472 (3,508) want: 20,480 K/sec
 


Tags:

drbd, solutionuh, handshake, protocol, conn, wfconnection, wfreportparams, asender, _receiver, alg, drbd_sync_handshake, aae, af, db, bf, cbea, cf, bits, flags, peer, uuid_compare, helper, sbin, drbdadm, initial, meta, networkfailure, terminated, terminating, detected, unresolved, disconnecting, receiving, reportstate, standalone, receiver, proc, api, proto, git, hash, ca, faacab, bc, bd, sighted, cs, ro, secondary, ds, uptodate, dunknown, ns, nr, dw, bm, pe, ua, ap, ep, wo, oos, module, userland, preferably, kernel, versions, config, disconnect, drbdsetup, ipv, defaults, sb, pri, discard, consensus, active, node, syncsource, primary, inconsistent, sync, rejoining, synctarget,

Latest Articles

  • How to allow SSH root user access in Linux/Debian/Mint/RHEL/Ubuntu/CentOS
  • Ansible Tutorial - Playbook How To Install From Scratch and Deploy LAMP + Wordpress on Remote Server
  • Ceph Install Errors on Proxmox / How To Fix Solution
  • Proxmox Update Error https://enterprise.proxmox.com/debian/pve bullseye InRelease 401 Unauthorized [IP: 144.217.225.162 443]
  • QEMU/KVM How to Hot-add A Virtual Disk .raw/.qcow2 via QEMU Monitor Commands
  • Proxmox How To Enable Ceph Distributed Storage Cluster with OSD and Pools
  • pulseaudio issue on QEMU/KVM guest VM when microphone is replugged/unplugged pulseaudio: pa_threaded_mainloop_lock failed pulseaudio: Reason: Invalid argument
  • Ubuntu Linux Mint - Volume Control Stopped Working
  • Proxmox Services Won't Start Failed to start The Proxmox VE cluster filesystem. Proxmox VE firewall. PVE Status Daemon. Proxmox VE scheduler. PVE Cluster HA Resource Manager Daemon. PVE Local HA Resource Manager Daemon.
  • Proxmox Guide FAQ / Errors / Howto
  • Virtualbox Vbox Issue Cannot Enable Nested Virtualization Button is Grayed/Greyed Out and Unclickable HowTo Solution
  • Virtualbox VBOX Howto Port Forward To Guests
  • Linux Ubuntu Debian Centos Mint - How To Check if Intel VT-x or AMD-V Hardware Virtualization is Enabled?
  • Linux Howto Zip Multiple Files and Directories
  • Windows Cannot Format USB drive Device Media is Write Protected Error Solution
  • Linux Mint 20 cannot install snapd missing solution
  • Virtualbox VBOX How To Install Guest-Utils/GuestUtils so drag and drop and clipboard works Ubuntu Mint Debian Linux
  • How to install Kubernetes with microk8s and deploy apps on Debian/Mint/Ubuntu Linux
  • vi how to delete everything to the end of the line or the rest of the line from the cursor
  • Cisco Howto Configure Console Port/Terminal/Comm Server with Async Cable Setup