DRBD Split-brain solution

Uh oh

[17925926.174277] block drbd0: Handshake successful: Agreed network protocol version 96
[17925926.174325] block drbd0: conn( WFConnection -> WFReportParams )
[17925926.174342] block drbd0: Starting asender thread (from drbd0_receiver [1682])
[17925926.174432] block drbd0: data-integrity-alg:
[17925926.174581] block drbd0: drbd_sync_handshake:
[17925926.174586] block drbd0: self 2AAE66AF9252D6DB:2815BF75CBEA0687:E303F24CF2826E13:E302F24CF2826E13 bits:73000 flags:0
[17925926.174591] block drbd0: peer 24F1983D580A1E83:2815BF75CBEA0686:E303F24CF2826E12:E302F24CF2826E13 bits:93874 flags:0
[17925926.174596] block drbd0: uuid_compare()=100 by rule 90
[17925926.174602] block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0
[17925926.180872] block drbd0: meta connection shut down by peer.
[17925926.181041] block drbd0: conn( WFReportParams -> NetworkFailure )
[17925926.181055] block drbd0: asender terminated
[17925926.181058] block drbd0: Terminating asender thread
[17925926.207698] block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0 exit code 0 (0x0)
[17925926.207704] block drbd0: Split-Brain detected but unresolved, dropping connection!
[17925926.207876] block drbd0: helper command: /sbin/drbdadm split-brain minor-0
[17925926.209653] block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)
[17925926.209659] block drbd0: conn( NetworkFailure -> Disconnecting )
[17925926.209668] block drbd0: error receiving ReportState, l: 4!
[17925926.209839] block drbd0: Connection closed
[17925926.209846] block drbd0: conn( Disconnecting -> StandAlone )
[17925926.209862] block drbd0: receiver terminated
[17925926.209865] block drbd0: Terminating receiver thread


[root@box18 ~]# cat /proc/drbd
version: 8.3.13 (api:88/proto:86-96)
GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by root@sighted, 2012-10-09 12:47:51
0: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown A r-----
ns:0 nr:0 dw:983961744 dr:1225316590 al:3645772 bm:110268 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:1619916
[root@box18 ~]# drbdadm connect r0
DRBD module version: 8.3.13
userland version: 8.3.16
preferably kernel and userland versions should match.
0: Failure: (125) Device has a net-config (use disconnect first)
Command 'drbdsetup 0 net ipv4:192.168.5.2.18:7788 ipv4:192.168.5.2.17:7788 A --set-defaults --create-device --after-sb-0pri=discard-least-changes --after-sb-1pri=consensus' terminated with exit code 10


[root@box18 ~]# drbdadm connect r0
DRBD module version: 8.3.13
userland version: 8.3.16
preferably kernel and userland versions should match.
0: Failure: (125) Device has a net-config (use disconnect first)
Command 'drbdsetup 0 net ipv4:192.168.5.2.18:7788 ipv4:192.168.5.2.17:7788 A --set-defaults --create-device --after-sb-0pri=discard-least-changes --after-sb-1pri=consensus' terminated with exit code 10

Solution


drbdadm disconnect r0

drbdadm -- --discard-my-data connect r0



#On the main active node


drbdadm connect r0
cat /proc/drbd
version: 8.3.13 (api:88/proto:86-96)
GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by root@sighted, 2012-10-09 12:47:51
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent A r-----
ns:3580 nr:0 dw:1597648248 dr:12347577 al:12090 bm:58295 lo:13 pe:169 ua:85 ap:1 ep:1 wo:b oos:3289420
[>....................] sync'ed: 0.2% (3289420/3290212)K
finish: 1:08:31 speed: 792 (792) K/sec


#on the dead node that is rejoining


cat /proc/drbd
version: 8.3.13 (api:88/proto:86-96)
GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by root@sighted, 2012-10-09 12:47:51
0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate A r-----
ns:0 nr:151088 dw:984106508 dr:1225316590 al:3645772 bm:110390 lo:61 pe:0 ua:2 ap:0 ep:3 wo:b oos:3170888
[>....................] sync'ed: 3.8% (3170888/3290212)K
finish: 0:11:48 speed: 4,472 (3,508) want: 20,480 K/sec


Tags:

drbd, solutionuh, handshake, protocol, conn, wfconnection, wfreportparams, asender, _receiver, alg, drbd_sync_handshake, aae, af, db, bf, cbea, cf, bits, flags, peer, uuid_compare, helper, sbin, drbdadm, initial, meta, networkfailure, terminated, terminating, detected, unresolved, disconnecting, receiving, reportstate, standalone, receiver, proc, api, proto, git, hash, ca, faacab, bc, bd, sighted, cs, ro, secondary, ds, uptodate, dunknown, ns, nr, dw, bm, pe, ua, ap, ep, wo, oos, module, userland, preferably, kernel, versions, config, disconnect, drbdsetup, ipv, defaults, sb, pri, discard, consensus, active, node, syncsource, primary, inconsistent, sync, rejoining, synctarget,

Latest Articles

  • Ubuntu Debian Linux Cannot Install Wine Solution - wine1.6 : Depends: wine1.6-i386 (= 1:1.6.2-0ubuntu14.2) but it is not installable wine1.4 : Depends: wine1.6 but it is not going to be installed
  • How To Install python 3.4 3.5 and up on Linux with wine - Working Solution
  • using Xvfb on virtual remote ssh server to have X graphical programs work
  • ssh Received disconnect from port 22:2: Too many authentication failures
  • named bind errors - DNSKEY: unable to find a DNSKEY which verifies the DNSKEY RRset and also matches a trusted key for '.'
  • OpenVZ vs LXC DIR mode poor security in LXC
  • httpd: Syntax error on line 221 of /etc/httpd/conf/httpd.conf: Syntax error on line 6 of /etc/httpd/conf.d/php.conf: Cannot load modules/libphp5.so into server: /lib64/libresolv.so.2: symbol __h_errno, version GLIBC_PRIVATE not defined in file libc.s
  • Radeon R3 GPU on Debian Crashing
  • MySQL 5.7 on Debian and Ubuntu - How To Reset Root Password
  • SSH and sshfs timeout settings keepalive
  • Linux How To Add User To Additional Group
  • Howto Set Static IP on boot in initramfs for dropbear or other purposes NFS, Linux, Debian, Ubuntu, CentOS
  • Convert and install to LUKS Encrypted Drive Ubuntu 18.04 19.10 Linux Mint and Debian Based Linux
  • Debian and Netplan
  • CentOS 8 how to restart the network!
  • CentOS 8 how to convert to a bootable mdadm RAID software array
  • ADATA USB Thumb Drive Issues
  • KMODE EXCEPTION NOT HANDLED - QEMU/KVM Won't Boot Windows 2016 or 10 Image or Physical Machine
  • Linux Mint / Ubuntu / Debian Mate Disable Guest Session and Hide Usernames on Lightdm Login screen GUI
  • SSH How To Create Public/Private Key Pair and with a Larger Keysize than 2048 bits