Kernel/make compilation time and how to improve compile times/compile the Linux kernel faster without hardware upgrades

I thought only a faster CPU and SSD would help but I already have a Quad-Core CPU and it wasn't being maxed out.  The actual tests were performed on an AMD-V enabled 128MB dual core VMWare container though.

There is a flag that can be passed to make in order to start multiple threads, by specifying 4 threads I was able to reduce the whole kernel compilation time from scratch by about 50%! (65minutes vs 31minutes!).  *Yes I did do a make clean before each compilation too!

*Part of the slow kernel time is that I use the slow method of making my own initramfs (not pre-compressed and the kernel compile takes like 10x longer for the same thing I could do with a script which I normally do).

Normal Make (single thread by default):

make

real    65m18.956s

Threaded Make (4 threads):

make -j 4

real    31m57.877s

second run:

real    27m28.745s
 

Threaded Make (8 threads):

*I believe the worse result is likely due to swapping since I only had 128MB of RAM.  Perhaps a lot more RAM could improve things too.

real    58m29.142s
user    33m3.616s
sys    19m13.064s

By increasing RAM to 512MB here are the results (when compiling RAM is more important than CPU and disk speed):

real    18m46.933s
user    20m11.776s
sys    6m51.334s

With 1GB of RAM

real    18m38.608s
user    20m31.857s
sys    7m46.141s

I believe the time was disappointing because of the initramfs creation.

With pre-created initramfs linked into kernel:

real    10m47.362s
user    18m24.837s
sys    1m48.095s
 

With 12 threads:

real    10m34.550s

Clearly the threads no longer help once the CPU is maxed out, I didn't check but considering with 8 threads that I was often at 80-90% CPU, now the CPU is the bottleneck.  I'm going to increase my cores to 4 and try again.

It only shaved off 13 seconds, but the crazy thing is that initramfs takes 8 minutes to create alone!  That's how inefficient the routine from the kernel is.  The same initramfs is created through a script in about 1 minute or less!


 

Snapshot of top with 8 threads showing high iowait:

You can really see iowait is starting to become a factor (40-60% on both cores on average).  I'm already running a RAID 1 with 7200 RPM 1TB drives.  I believe SSD would make a huge improvement with the iowait.  The CPU io often hits 70-80% but I believe the main culprit is the high iowait.  The system with 8 threads is quite unresponsive to even shell commands and typing.

11:24:56 up 54 days,  1:29,  5 users,  load average: 11.80, 10.36, 6.08
 

top - 11:22:12 up 54 days,  1:27,  5 users,  load average: 11.79, 9.53, 5.04
Tasks: 121 total,   6 running,  97 sleeping,  18 stopped,   0 zombie
 Cpu0 : 18.1% us, 13.3% sy,  0.0% ni,  0.0% id, 67.7% wa,  0.0% hi,  0.9% si
 Cpu1 : 13.3% us, 19.5% sy,  0.0% ni,  0.0% id, 53.6% wa,  1.1% hi, 12.4% si
Mem:    126980k total,   116308k used,    10672k free,      788k buffers
Swap:   377488k total,   107036k used,   270452k free,     5180k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                        
 7970 root      18   0 17596  14m 2976 D 18.2 11.5   0:00.75 cc1                                                                                            
 7932 root      18   0 19220  13m 1612 D 17.8 10.9   0:00.94 cc1                                                                                            
 7923 root      18   0 24152  13m 3140 R 13.4 11.0   0:01.37 cc1                                                                                            
 7938 root      18   0 19448  12m 1612 R 10.8 10.4   0:00.83 cc1                                                                                            
   14 root      10  -5     0    0    0 S  8.0  0.0   5:17.26 kblockd/1                                                                                      
 7687 root      18   0 36760  13m 3036 R  7.6 10.7   0:05.45 cc1                                                                                            
 7905 root      18   0 21188  14m 3064 D  7.0 11.5   0:00.64 cc1                                                                                            
  118 root      10  -5     0    0    0 D  3.5  0.0  15:23.71 kswapd0                                                                                        
 7915 root      18   0 27164  17m 1616 D  2.9 14.1   0:00.69 cc1                 

With 512MB of RAM instead of 128MB

real    18m46.933s
user    20m11.776s
sys    6m51.334s
 

Things don't fee lagged at all on the system unlike last time when it had 128MB of RAM.

The load is lower and iowait is virtually non-existent.

 03:52:52 up 4 min,  2 users,  load average: 10.55, 5.14, 2.02

 04:02:46 up 14 min,  2 users,  load average: 1.85, 5.12, 4.13

top - 03:53:11 up 5 min,  2 users,  load average: 10.30, 5.43, 2.18
Tasks:  93 total,  12 running,  81 sleeping,   0 stopped,   0 zombie
 Cpu0 : 90.0% us,  9.7% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.3% hi,  0.0% si
 Cpu1 : 89.3% us, 10.0% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.7% si
Mem:    516820k total,   431108k used,    85712k free,   176120k buffers
Swap:   377488k total,        0k used,   377488k free,   111164k cached

top - 04:02:42 up 14 min,  2 users,  load average: 1.84, 5.18, 4.14
Tasks:  55 total,   2 running,  53 sleeping,   0 stopped,   0 zombie
 Cpu0 : 10.0% us, 40.2% sy,  0.0% ni, 49.8% id,  0.0% wa,  0.0% hi,  0.0% si
 Cpu1 : 11.6% us, 37.9% sy,  0.0% ni, 49.5% id,  1.0% wa,  0.0% hi,  0.0% si
 

Free Memory gets low sometimes:

            total       used       free     shared    buffers     cached
Mem:           504        486         18          0          4        427
 

With 1GB

Clearly 1GB is the sweet spot, I'm tempted to turn the threads up from 8 to at least 12 or 16.

We can also see that CPU usage gets higher, so it is a factor and that iowait when compiling is usually caused by swapping because of too little RAM.

 Cpu0 : 92.0% us,  8.0% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.0% si
 Cpu1 : 93.4% us,  6.6% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.0% si

total       used       free     shared    buffers     cached
Mem:          1012        408        603          0         37        291

 

 04:24:24 up 13 min,  2 users,  load average: 2.43, 6.33, 4.47

 04:33:13 up 21 min,  2 users,  load average: 10.58, 7.00, 5.06

Even with the high load the system is very responsive, unlike at 128MB of RAM

Free Mem does get low still:

             total       used       free     shared    buffers     cached
Mem:          1012        986         26          0          7        900
-/+ buffers/cache:         77        934
Swap:          368          0        368 

*Note that if you specify -j with no number it opens unlimited threads, it basically causes gcc to crash/fail in my experience.  Perhaps with more memory this wouldn't have happened, I'm not sure what caused it for sure other than my system being unable to handle unlimited threads.

You'll get errors like this if specifying unlimited threads:


  CC      kernel/time/timekeeping.o
gcc: gcc: Internal error: Killed (program cc1)
Please submit a full bug report.
See <URL:http://gcc.gnu.org/bugs.html> for instructions.
For Debian GNU/Linux specific bug reporting instructions, see
<URL:file:///usr/share/doc/gcc-3.4/README.Bugs>.

Internal error: Killed (program cc1)
Please submit a full bug report.
See <URL:http://gcc.gnu.org/bugs.html> for instructions.
For Debian GNU/Linux specific bug reporting instructions, see
<URL:file:///usr/share/doc/gcc-3.4/README.Bugs>.

make[2]: make[1]: *** [arch/x86/kernel/setup.o] Killed
*** [fs/file_table.o] Killed
gcc: gcc: make[2]: *** [arch/x86/kernel/x86_init.o] Killed
make[1]: *** [fs/super.o] Killed
gcc: gcc: Internal error: Killed (program cc1)
Please submit a full bug report.
See <URL:http://gcc.gnu.org/bugs.html> for instructions.
For Debian GNU/Linux specific bug reporting instructions, see
<URL:file:///usr/share/doc/gcc-3.4/README.Bugs>.
Internal error: Killed (program cc1)
Please submit a full bug report.
See <URL:http://gcc.gnu.org/bugs.html> for instructions.
For Debian GNU/Linux specific bug reporting instructions, see
<URL:file:///usr/share/doc/gcc-3.4/README.Bugs>.
Internal error: Killed (program cc1)
Please submit a full bug report.
See <URL:http://gcc.gnu.org/bugs.html> for instructions.
For Debian GNU/Linux specific bug reporting instructions, see
<URL:file:///usr/share/doc/gcc-3.4/README.Bugs>.
Internal error: Killed (program cc1)
Please submit a full bug report.
See <URL:http://gcc.gnu.org/bugs.html> for instructions.
For Debian GNU/Linux specific bug reporting instructions, see
<URL:file:///usr/share/doc/gcc-3.4/README.Bugs>.
make[2]:

 

Conclusion/What I Learned

The single most important factor for faster compiling is RAM and 1GB+ is preferable.  The first wall I hit was high iowait due to insufficient RAM and swapping.  With more RAM the iowait virtually disappears and you can see the CPU's getting loaded to 80-90%.

Basically using make with more threads, decreases the compile time exponentially, but only so long as you have enough RAM to support those threads, and the next bottleneck will become CPU processing power.  The key is to add more cores at that point. Having a Quad Core or even 6 core with lots of RAM would give you the best performance and faster compliing.  I feel disk speed is of little impact when compiling so SSD wouldn't make much of a difference.


Tags:

kernel, compilation, improve, compile, linux, hardware, upgradesi, cpu, ssd, quad, wasn, maxed, performed, amd, enabled, mb, dual, vmware, container, multiple, threads, specifying, reduce, vs, method, initramfs, compressed, default, threaded, swapping, ram, user, sys, increasing, compiling, disk, gb, disappointing, creation, linked, didn, bottleneck, cores, shaved, inefficient, snapshot, iowait, factor, raid, rpm, tb, improvement, io, culprit, unresponsive, shell, commands, typing, users, tasks, sy, ni, wa, mem, buffers, swap, cached, pid, pr, virt, res, shr, cc, kblockd, kswapd, fee, lagged, virtually, existent, min, tempted, usage, responsive, cache, specify, unlimited, gcc, wouldn, unable, ll, errors, timekeeping, submit, url, http, gnu, org, html, debian, reporting, usr, readme, fs, file_table, _init, preferable, insufficient, disappears, decreases, exponentially, processing, compliing,

Latest Articles

  • Cloned VM/Server/Computer in Linux won't boot and goes to initramfs busybox Solution
  • How To Add Windows 7 8 10 11 to GRUB Boot List Dual Booting
  • How to configure OpenDKIM on Linux with Postfix and setup bind zonefile
  • Debian Ubuntu 10/11/12 Linux how to get tftpd-hpa server setup tutorial
  • efibootmgr: option requires an argument -- 'd' efibootmgr version 15 grub-install.real: error: efibootmgr failed to register the boot entry: Operation not permitted.
  • Apache Error Won't start SSL Cert Issue Solution Unable to configure verify locations for client authentication SSL Library Error: 151441510 error:0906D066:PEM routines:PEM_read_bio:bad end line SSL Library Error: 185090057 error:0B084009:x509 certif
  • Linux Debian Mint Ubuntu Bridge br0 gets random IP
  • redis requirements
  • How to kill a docker swarm
  • docker swarm silly issues
  • isc-dhcp-server dhcpd how to get longer lease
  • nvidia cannot resume from sleep Comm: nvidia-sleep.sh Tainted: Linux Ubuntu Mint Debian
  • zfs and LUKS how to recover in Linux
  • [error] (28)No space left on device: Cannot create SSLMutex Apache Solution Linux CentOS Ubuntu Debian Mint
  • Save money on bandwidth by disabling reflective rpc queries in Linux CentOS RHEL Ubuntu Debian
  • How to access a disk with bad superblock Linux Ubuntu Debian Redhat CentOS ext3 ext4
  • ImageMagick error convert solution - convert-im6.q16: cache resources exhausted
  • PTY allocation request failed on channel 0 solution
  • docker error not supported as upperdir failed to start daemon: error initializing graphdriver: driver not supported
  • Migrated Linux Ubuntu Mint not starting services due to broken /var/run and dbus - Failed to connect to bus: No such file or directory solution