DRBD Slow Performance - 99.99 % [jbd2/drbd0-8] highiowait solution -

DRBD Slow Performance - 99.99 % [jbd2/drbd0-8] highiowait solution

Tired of checking iotop and seeing that your drbd partition is using 99.99% of io all the time and finding your drbd device performs slow in general?

This is especially an issue in versions of DRBD in the 8.3 tree in particular one documented case is on "8.3.13" but it likely applies to other devices.

The symptoms are that resyncing is fine and normal but any reasonable amount of activity is very slow and lagged and creates a high server load and consequently high io wait.  You may not notice it though until you apply a reasonable load or usage to the server and DRBD device.

In some kernels and some versions you may get the following error in dmesg

block drbd0: [drbd0_worker/1670] sock_sendmsg time expired, ko = 4294967295

*But we have seen many cases where the above error is not present (perhaps an older kernel or module does not recognize it is being blocked).

Here are some quick sysctl.conf kernel tuning tips that have changed a server's load from 10-18, to less than 1.

Add or edit these in sysctl.conf to solve the issue

net.ipv4.tcp_rmem = 131072  131072  10485760
net.ipv4.tcp_wmem = 131072  131072  10485760
vm.dirty_ratio = 10
vm.dirty_background_ratio = 4

 

This is a preferable way rather than hacking or changing DRBD settings by upgrading the userland or kernel manually or while it runs (results can be unpredictable) whereas this kernel tuning has virtually no risk or impact on DRBD except a positive one.


  • bash Linux how to get first or last letters of a word
  • l2tp ipsec VPN Error Sep 12 18:16:25 vps pluto[7299]: ERROR: asynchronous network error report on eth0 (sport=500) for message to 192.5.6.2 port 20640, complainant 192.5.6.2: Connection refused [errno 111, origin ICMP type 3 code 3 (not authenticated
  • Centos 5.9 Working Vault Repo file
  • Disable SSH Password Authentication to Increase Security and Harden SSH Linux Unix Server Ubuntu Mint Centos Debian
  • ecryptfs how to mount or recover from a backup
  • Linux bash script to see what connected computers respond to ping
  • rsync specify alternate port non-standard port than 22
  • Centos 6.6/6.9 KVM VM Kernel Panic On Boot - Kernel panic - not syncing: Attempted to kill init!
  • Cannot allocate TUN/TAP dev dynamically - FreeBSD/OpenBSD OpenVPN Client error Solution
  • Linux Mint/Ubuntu/Debian apt how to downgrade a package
  • Ubuntu Linux Mint How To Exclude Stop Package from being Upgraded or Installed
  • Intel NUC Lower Power Green Computing Boxes Review/Comparison of J3160 and J3455 Models
  • How to backup entire computer Linux Mint Ubuntu with tar
  • tar how to ignore failed reads and not exit
  • kdenlive titles/text renders as white screen when using .sh script
  • [1035724.274610] [drm:intel_pipe_update_end [i915_bpo]] *ERROR* Atomic update failure on pipe A (start=62076478 end=62076479) time 102 us, min 894, max 899, scanline start 893, end 900 W: Possible missing firmware /lib/firmware/i915/kbl_dmc_ver1.bin
  • ffmpeg how to watermark and concatenate in one command
  • /usr/bin/ld: cannot find -lboost_system-mt-s /usr/bin/ld: cannot find -lboost_filesystem-mt-s /usr/bin/ld: cannot find -lboost_program_options-mt-s /usr/bin/ld: cannot find -lboost_thread-mt-s collect2: error: ld returned 1 exit status make: *** [cag
  • Wine uninstalled broken on Linux Mint
  • ffmpeg trouble concatenating similar but different files