DRBD Slow Performance - 99.99 % [jbd2/drbd0-8] highiowait solution -

DRBD Slow Performance - 99.99 % [jbd2/drbd0-8] highiowait solution

Tired of checking iotop and seeing that your drbd partition is using 99.99% of io all the time and finding your drbd device performs slow in general?

This is especially an issue in versions of DRBD in the 8.3 tree in particular one documented case is on "8.3.13" but it likely applies to other devices.

The symptoms are that resyncing is fine and normal but any reasonable amount of activity is very slow and lagged and creates a high server load and consequently high io wait.  You may not notice it though until you apply a reasonable load or usage to the server and DRBD device.

In some kernels and some versions you may get the following error in dmesg

block drbd0: [drbd0_worker/1670] sock_sendmsg time expired, ko = 4294967295

*But we have seen many cases where the above error is not present (perhaps an older kernel or module does not recognize it is being blocked).

Here are some quick sysctl.conf kernel tuning tips that have changed a server's load from 10-18, to less than 1.

Add or edit these in sysctl.conf to solve the issue

net.ipv4.tcp_rmem = 131072  131072  10485760
net.ipv4.tcp_wmem = 131072  131072  10485760
vm.dirty_ratio = 10
vm.dirty_background_ratio = 4

 

This is a preferable way rather than hacking or changing DRBD settings by upgrading the userland or kernel manually or while it runs (results can be unpredictable) whereas this kernel tuning has virtually no risk or impact on DRBD except a positive one.


  • named[1525]: error (broken trust chain) resolving 'min-api.cryptocompare.com/A/IN': 173.245.58.78#53 solution
  • Linux Ubuntu Mint Debian cannot play xvid,x264,mp4 or any videos codec issue - xplayer Could not initialize supporting library.
  • wget howto output to terminal or script instead of file
  • imagemagick convert to chop off top of image
  • How To Set or Change Useragent in Firefox and Seamonkey
  • How To Turn Any .iso file into a bootable USB drive
  • Linux Mint Ubuntu Debian How To Disable Webcam Automatically Onboot To Prevent Spying and Privacy Violations
  • IcedTea Java Web Viewer Config Utility How To Enter
  • Debian Linux Mint Ubuntu Disable Automatic apt-get update
  • How To Install Seamonkey Web Browser on Debian Ubuntu Linux Mint
  • Prevent SSH Bruteforce and Hacks By Disabling Password Authentication
  • SMF Forums / Simple Machines Forums Not Displaying Images Theme or Styles Properly using 127.0.0.1
  • solution mysqldump: Got error: 1044: Access denied for user 'user'@'localhost' to database 'thedb' when using LOCK TABLES
  • MySQL How To Grant Access To ALL Databases For Export and Backup Purposes
  • mdadm how to stop or start a check
  • vzquota : (error) Quota on syscall for id 4532: No such file or directory vzquota on failed [3] OpenVZ Error and Solution
  • Apache htaccess Custom ErrorDocument not working properly for root home page 403 Error Issue and Solution
  • syslinux / pxelinux how to boot from local drive how to
  • samba how to listen on specific IP only
  • How To Install Windows Server 7 8 10 12 2008 2012 2014 2016 Servers Desktops using Linux tftp, dhcpd and samba