DRBD Slow Performance - 99.99 % [jbd2/drbd0-8] highiowait solution

Tired of checking iotop and seeing that your drbd partition is using 99.99% of io all the time and finding your drbd device performs slow in general?

This is especially an issue in versions of DRBD in the 8.3 tree in particular one documented case is on "8.3.13" but it likely applies to other devices.

The symptoms are that resyncing is fine and normal but any reasonable amount of activity is very slow and lagged and creates a high server load and consequently high io wait.  You may not notice it though until you apply a reasonable load or usage to the server and DRBD device.

In some kernels and some versions you may get the following error in dmesg

block drbd0: [drbd0_worker/1670] sock_sendmsg time expired, ko = 4294967295

*But we have seen many cases where the above error is not present (perhaps an older kernel or module does not recognize it is being blocked).

Here are some quick sysctl.conf kernel tuning tips that have changed a server's load from 10-18, to less than 1.

Add or edit these in sysctl.conf to solve the issue

net.ipv4.tcp_rmem = 131072  131072  10485760
net.ipv4.tcp_wmem = 131072  131072  10485760
vm.dirty_ratio = 10
vm.dirty_background_ratio = 4

 

This is a preferable way rather than hacking or changing DRBD settings by upgrading the userland or kernel manually or while it runs (results can be unpredictable) whereas this kernel tuning has virtually no risk or impact on DRBD except a positive one.


Tags:

drbd, jbd, highiowait, solutiontired, iotop, partition, io, performs, versions, documented, quot, applies, devices, resyncing, lagged, creates, server, consequently, usage, kernels, dmesg, _worker, sock_sendmsg, expired, ko, kernel, module, blocked, sysctl, conf, tuning, edit, ipv, tcp_rmem, tcp_wmem, vm, dirty_ratio, dirty_background_ratio, preferable, hacking, settings, upgrading, userland, manually, unpredictable, whereas, virtually,

Latest Articles

  • Huion and Wacom Tablets How To Install in Linux Mint / Ubuntu and make the stylus work properly
  • ffmpeg how to cut certain parts of video out
  • ffmpeg how to concat and join two video clips
  • mencoder instead of ffmpeg to join or concatenate video files with different audio streams
  • Linux How To Stop Missing Drive from Halting Boot Process in fstab
  • How To Replace Audio Track of Video using ffmpeg
  • qemu-img convert formats vdi vmdk raw qcow2
  • Linux and Windows Dual Boot Crazy Time Issues
  • dynagen / dynamips 100% high CPU usage solution - how to set the idlepc value
  • How To Setup a Cisco CME (Cisco Manager Express) Virtual Router under Linux using dynamips and dynagen
  • Linux Mint Ubuntu Debian CentOS Dual Boot Install Issues
  • Linux Mint Ubuntu Debian Centos RHEL no sound solution
  • Linux Mint/Debian/Ubuntu/Centos Installer black grub screen and blank screen after trying to boot installer or main OS
  • Linux Mint Dual Boot Install Avoid Wiping our your Main C: drive /dev/sda MBR and EFI
  • QEMU-KVM soundhw deprecated how to enable sound in QEMU 4.x series
  • Virtualbox Error Cannot register the hard disk because a hard disk with UUID already exists solution
  • kernel: [549267.368859] mate-terminal[7871]: segfault at 2000000101 ip 00007f5d0a9548f0 sp 00007fff7012c610 error 4 in libgobject-2.0.so.0.4800.2[7f5d0a920000+52000]
  • apcupsd how to setup and monitor APC UPS units
  • How To Password Reset, Recover, Bypass, Remove and Unlock on Windows 10,8,7,Vista,XP,NT,2000,2003,2008,2012,2016,2019 Administrative Login Programs
  • Nvidia Ubuntu Linux Screentearing Video with solution driver