DRBD Slow Performance - 99.99 % [jbd2/drbd0-8] highiowait solution -

DRBD Slow Performance - 99.99 % [jbd2/drbd0-8] highiowait solution

Tired of checking iotop and seeing that your drbd partition is using 99.99% of io all the time and finding your drbd device performs slow in general?

This is especially an issue in versions of DRBD in the 8.3 tree in particular one documented case is on "8.3.13" but it likely applies to other devices.

The symptoms are that resyncing is fine and normal but any reasonable amount of activity is very slow and lagged and creates a high server load and consequently high io wait.  You may not notice it though until you apply a reasonable load or usage to the server and DRBD device.

In some kernels and some versions you may get the following error in dmesg

block drbd0: [drbd0_worker/1670] sock_sendmsg time expired, ko = 4294967295

*But we have seen many cases where the above error is not present (perhaps an older kernel or module does not recognize it is being blocked).

Here are some quick sysctl.conf kernel tuning tips that have changed a server's load from 10-18, to less than 1.

Add or edit these in sysctl.conf to solve the issue

net.ipv4.tcp_rmem = 131072  131072  10485760
net.ipv4.tcp_wmem = 131072  131072  10485760
vm.dirty_ratio = 10
vm.dirty_background_ratio = 4

 

This is a preferable way rather than hacking or changing DRBD settings by upgrading the userland or kernel manually or while it runs (results can be unpredictable) whereas this kernel tuning has virtually no risk or impact on DRBD except a positive one.


  • /usr/bin/ld: cannot find -lboost_system-mt-s /usr/bin/ld: cannot find -lboost_filesystem-mt-s /usr/bin/ld: cannot find -lboost_program_options-mt-s /usr/bin/ld: cannot find -lboost_thread-mt-s collect2: error: ld returned 1 exit status make: *** [cag
  • Wine uninstalled broken on Linux Mint
  • ffmpeg trouble concatenating similar but different files
  • ffmpeg Unable to Use Hardware Encoding with Nvidia 3.40 Driver and GT210 card
  • Linux Mint USB Kernel Tainted and Locked Port/Dev File
  • ffmpeg Linux Mint download, compile and install howto
  • OpenVZ error : Container start failed (try to check kernel messages, e.g. "dmesg | tail") Locked by: pid 166638, cmdline vzctl start 888171
  • How to extract view contents of initramfs image gzip'd
  • Linux how to copy GPT partition table with dd
  • Centos 7 How To Change Hostname
  • Debian/Mint/Ubuntu Font Packs/Recommended To Install
  • Migrate Centos 7 from Single HDD to mdadm RAID 10 array:
  • How to change reserved blocks in Linux partition
  • Reading package lists... Done W: GPG error: http://ppa.launchpad.net trusty InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY D46F45428842CE5E
  • Centos 7 A start job is running for dev-mapper-clx2droot.device (8min 44s / no limit)
  • USB 3.0 External HDD Enclosure Seagate UAS problems - [sdd] tag#1 CDB: Write(16) 8a 00 00 00 00 01 70 04 08 68 00 00 00 08 00 00
  • Centos 7 Cudaminer Nvidia setup guide
  • USB 3.0 PCI x1 Card Review VIA VL805 on Linux Review and Experience
  • rsync run as root sudo without password
  • Why won't my Linux Mint boot after I manually installed a new kernel?