Linux Out of Memory OOM Object Killer Solution "Out of memory: kill process 1955 (sshd) score 81 or a child"

I had a system running a 128MB live CD image with 2.8 gigs of available RAM and the OOM kernel killer went crazy when using dd for more than 8 minutes and kept killing everything.  I've read that this is due to a low-memory issue and paging in the kernel and 32-bit systems with lots of RAM.

I even enabled swapspace on my LiveCD and the issue happened 25 minutes into dd rather than 8 minutes, so what gives?
Also no swap space was ever used!

cat /proc/swaps 
Filename                Type        Size    Used    Priority
/dev/sda2                               partition    30716152    2784    -1


Here is a nice thread on the issue: http://www.redhat.com/archives/redhat-list/2007-August/msg00060.html
I believe the option below in the kernel under "Processor Type & Features" should resolve this.

Solution for OOM


 [*] Allocate 3rd-level pagetables from highmem


Here is meminfo when the OOM killer is not active

-bash-3.1# cat /proc/meminfo
MemTotal:      2855128 kB
MemFree:         77016 kB
Buffers:         47012 kB
Cached:        2646720 kB
SwapCached:        908 kB
Active:        1411204 kB
Inactive:      1283876 kB
HighTotal:     1963904 kB
HighFree:          428 kB
LowTotal:       891224 kB
LowFree:         76588 kB
SwapTotal:    30716152 kB
SwapFree:     30712628 kB
Dirty:               0 kB
Writeback:         296 kB
AnonPages:         876 kB
Mapped:           1424 kB
Slab:            45668 kB
SReclaimable:    40420 kB
SUnreclaim:       5248 kB
PageTables:        328 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
WritebackTmp:        0 kB
CommitLimit:  32143716 kB
Committed_AS:     8416 kB
VmallocTotal:   114680 kB
VmallocUsed:     16336 kB
VmallocChunk:    97976 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     4096 kB
DirectMap4k:      4096 kB
DirectMap4M:    913408 kB

Here is meminfo when OOM killer is active
 

cat /proc/meminfo
MemTotal:      2855128 kB
MemFree:         76440 kB
Buffers:             0 kB
Cached:        2712884 kB
SwapCached:        520 kB
Active:        1433708 kB
Inactive:      1279804 kB
HighTotal:     1963904 kB
HighFree:          428 kB
LowTotal:       891224 kB
LowFree:         76012 kB
SwapTotal:    30716152 kB
SwapFree:     30713364 kB
Dirty:               0 kB
Writeback:           0 kB
AnonPages:         372 kB
Mapped:           2616 kB
Slab:            28280 kB
SReclaimable:    22952 kB
SUnreclaim:       5328 kB
PageTables:        320 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
WritebackTmp:        0 kB
CommitLimit:  32143716 kB
Committed_AS:     6888 kB
VmallocTotal:   114680 kB
VmallocUsed:     16336 kB
VmallocChunk:    97976 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     4096 kB
DirectMap4k:      4096 kB
DirectMap4M:    913408 kB

dmesg

2kB
720352 pages RAM
490976 pages HighMem
6570 pages reserved
1585 pages shared
692675 pages non-shared
Out of memory: kill process 1955 (sshd) score 81 or a child
Killed process 3302 (sshd)
bash invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0
Pid: 3311, comm: bash Not tainted 2.6.27.54 #31
 [<c014ecd2>] oom_kill_process+0xe6/0x10c
 [<c014efd3>] out_of_memory+0xaf/0x100
 [<c0150844>] __alloc_pages_internal+0x31c/0x3b0
 [<c015a53f>] do_wp_page+0x9f/0x49c
 [<c011855e>] ? task_new_fair+0x6e/0xb0
 [<c015b757>] handle_mm_fault+0x183/0x1e0
 [<c0113e07>] do_page_fault+0x11f/0x43c
 [<c0113ce8>] ? do_page_fault+0x0/0x43c
 [<c026f572>] error_code+0x72/0x78
 =======================
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd: 150
CPU    1: hi:  186, btch:  31 usd: 171
CPU    2: hi:  186, btch:  31 usd: 145
CPU    3: hi:  186, btch:  31 usd: 153
HighMem per-cpu:
CPU    0: hi:  186, btch:  31 usd:  80
CPU    1: hi:  186, btch:  31 usd: 157
CPU    2: hi:  186, btch:  31 usd: 208
CPU    3: hi:  186, btch:  31 usd: 137
Active:358187 inactive:320084 dirty:0 writeback:0 unstable:0
 free:19121 slab:7075 mapped:659 pagetables:82 bounce:0
DMA free:11164kB min:64kB low:80kB high:96kB active:1472kB inactive:1240kB present:15868kB pages_scanned:9835 all_unreclaimable? yes
lowmem_reserve[]: 0 873 2776 2776
Normal free:64892kB min:3744kB low:4680kB high:5616kB active:382712kB inactive:382612kB present:894080kB pages_scanned:372896 all_unreclaimable? no
lowmem_reserve[]: 0 0 15223 15223
HighMem free:428kB min:512kB low:2552kB high:4592kB active:1048564kB inactive:896484kB present:1948560kB pages_scanned:31617081 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 7*8kB 40*16kB 75*32kB 36*64kB 9*128kB 4*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 11164kB
Normal: 15423*4kB 18*8kB 9*16kB 2*32kB 6*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 64988kB
HighMem: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 428kB
678222 total pagecache pages
28 pages in swap cache
Swap cache stats: add 9762, delete 9734, find 2155/3264
Free swap  = 30713264kB
Total swap = 30716152kB
720352 pages RAM
490976 pages HighMem
6570 pages reserved
1911 pages shared
692514 pages non-shared
Out of memory: kill process 1955 (sshd) score 81 or a child
Killed process 3310 (sshd)
cronscript.sh invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0
Pid: 3322, comm: cronscript.sh Not tainted 2.6.27.54 #31
 [<c014ecd2>] oom_kill_process+0xe6/0x10c
 [<c014efd3>] out_of_memory+0xaf/0x100
 [<c0150844>] __alloc_pages_internal+0x31c/0x3b0
 [<c015a53f>] do_wp_page+0x9f/0x49c
 [<c0132948>] ? __wake_up_bit+0xc/0x40
 [<c015b757>] handle_mm_fault+0x183/0x1e0
 [<c0113e07>] do_page_fault+0x11f/0x43c
 [<c0113ce8>] ? do_page_fault+0x0/0x43c
 [<c026f572>] error_code+0x72/0x78
 =======================
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd: 143
CPU    1: hi:  186, btch:  31 usd: 161
CPU    2: hi:  186, btch:  31 usd: 142
CPU    3: hi:  186, btch:  31 usd: 171
HighMem per-cpu:
CPU    0: hi:  186, btch:  31 usd:  80
CPU    1: hi:  186, btch:  31 usd: 157
CPU    2: hi:  186, btch:  31 usd: 208
CPU    3: hi:  186, btch:  31 usd: 137
Active:358354 inactive:319939 dirty:0 writeback:0 unstable:0
 free:19071 slab:7075 mapped:689 pagetables:111 bounce:0
DMA free:11164kB min:64kB low:80kB high:96kB active:1472kB inactive:1240kB present:15868kB pages_scanned:10539 all_unreclaimable? yes
lowmem_reserve[]: 0 873 2776 2776
Normal free:64692kB min:3744kB low:4680kB high:5616kB active:383072kB inactive:382340kB present:894080kB pages_scanned:1310990 all_unreclaimable? yes
lowmem_reserve[]: 0 0 15223 15223
HighMem free:428kB min:512kB low:2552kB high:4592kB active:1048872kB inactive:896176kB present:1948560kB pages_scanned:31666018 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 7*8kB 40*16kB 75*32kB 36*64kB 9*128kB 4*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 11164kB
Normal: 15343*4kB 21*8kB 9*16kB 2*32kB 6*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 64692kB
HighMem: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 428kB
678265 total pagecache pages
61 pages in swap cache
Swap cache stats: add 11379, delete 11318, find 2543/3850
Free swap  = 30712744kB
Total swap = 30716152kB
720352 pages RAM
490976 pages HighMem
6570 pages reserved
2492 pages shared
692547 pages non-shared
Out of memory: kill process 3318 (cronscript.sh) score 107 or a child
Killed process 3319 (cronscript.sh)
bash invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0
Pid: 3323, comm: bash Not tainted 2.6.27.54 #31
 [<c014ecd2>] oom_kill_process+0xe6/0x10c
 [<c014efd3>] out_of_memory+0xaf/0x100
 [<c0150844>] __alloc_pages_internal+0x31c/0x3b0
 [<c015a53f>] do_wp_page+0x9f/0x49c
 [<c015b757>] handle_mm_fault+0x183/0x1e0
 [<c0113e07>] do_page_fault+0x11f/0x43c
 [<c0113ce8>] ? do_page_fault+0x0/0x43c
 [<c026f572>] error_code+0x72/0x78
 =======================
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd: 178
CPU    1: hi:  186, btch:  31 usd: 180
CPU    2: hi:  186, btch:  31 usd: 171
CPU    3: hi:  186, btch:  31 usd: 179
HighMem per-cpu:
CPU    0: hi:  186, btch:  31 usd:  80
CPU    1: hi:  186, btch:  31 usd: 157
CPU    2: hi:  186, btch:  31 usd: 208
CPU    3: hi:  186, btch:  31 usd: 137
Active:358137 inactive:320124 dirty:0 writeback:0 unstable:0
 free:19044 slab:7075 mapped:659 pagetables:100 bounce:0
DMA free:11164kB min:64kB low:80kB high:96kB active:1464kB inactive:1240kB present:15868kB pages_scanned:10731 all_unreclaimable? yes
lowmem_reserve[]: 0 873 2776 2776
Normal free:64584kB min:3744kB low:4680kB high:5616kB active:382404kB inactive:382720kB present:894080kB pages_scanned:1952385 all_unreclaimable? no
lowmem_reserve[]: 0 0 15223 15223
HighMem free:428kB min:512kB low:2552kB high:4592kB active:1048680kB inactive:896424kB present:1948560kB pages_scanned:31674398 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 7*8kB 40*16kB 75*32kB 36*64kB 9*128kB 4*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 11164kB
Normal: 15332*4kB 13*8kB 9*16kB 2*32kB 6*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 64584kB
HighMem: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 428kB
678229 total pagecache pages
14 pages in swap cache
Swap cache stats: add 11572, delete 11558, find 2617/3965
Free swap  = 30712912kB
Total swap = 30716152kB
720352 pages RAM
490976 pages HighMem
6570 pages reserved
2337 pages shared
692510 pages non-shared
Out of memory: kill process 1955 (sshd) score 81 or a child
Killed process 3313 (sshd)
sshd invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0
Pid: 1955, comm: sshd Not tainted 2.6.27.54 #31
 [<c014ecd2>] oom_kill_process+0xe6/0x10c
 [<c014efd3>] out_of_memory+0xaf/0x100
 [<c0150844>] __alloc_pages_internal+0x31c/0x3b0
 [<c0162251>] read_swap_cache_async+0x81/0xb4
 [<c01622d0>] swapin_readahead+0x4c/0x84
 [<c015afc9>] do_swap_page+0x16d/0x254
 [<c015b73c>] handle_mm_fault+0x168/0x1e0
 [<c0113e07>] do_page_fault+0x11f/0x43c
 [<c0113ce8>] ? do_page_fault+0x0/0x43c
 [<c026f572>] error_code+0x72/0x78
 [<c01cb286>] ? __copy_to_user_ll+0x46/0x178
 [<c0102a9f>] setup_frame+0x10b/0x324
 [<c0129e7d>] ? dequeue_signal+0x29/0x114
 [<c012b4a6>] ? ptrace_signal+0xa/0xbc
 [<c0103266>] handle_signal+0x116/0x15c
 [<c0103322>] do_signal+0x76/0x128
 [<c015af66>] ? do_swap_page+0x10a/0x254
 [<c017d4f6>] ? sys_select+0x9e/0x15c
 [<c01033f5>] do_notify_resume+0x21/0x24
 [<c01035b6>] work_notifysig+0x13/0x19
 =======================
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd: 183
CPU    1: hi:  186, btch:  31 usd: 180
CPU    2: hi:  186, btch:  31 usd: 171
CPU    3: hi:  186, btch:  31 usd: 179
HighMem per-cpu:
CPU    0: hi:  186, btch:  31 usd:  80
CPU    1: hi:  186, btch:  31 usd: 157
CPU    2: hi:  186, btch:  31 usd: 208
CPU    3: hi:  186, btch:  31 usd: 137
Active:358256 inactive:320033 dirty:0 writeback:0 unstable:0
 free:19044 slab:7075 mapped:613 pagetables:100 bounce:0
DMA free:11164kB min:64kB low:80kB high:96kB active:1464kB inactive:1240kB present:15868kB pages_scanned:10731 all_unreclaimable? yes
lowmem_reserve[]: 0 873 2776 2776
Normal free:64584kB min:3744kB low:4680kB high:5616kB active:382836kB inactive:382456kB present:894080kB pages_scanned:1975428 all_unreclaimable? yes
lowmem_reserve[]: 0 0 15223 15223
HighMem free:428kB min:512kB low:2552kB high:4592kB active:1048724kB inactive:896436kB present:1948560kB pages_scanned:31675298 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 7*8kB 40*16kB 75*32kB 36*64kB 9*128kB 4*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 11164kB
Normal: 15332*4kB 13*8kB 9*16kB 2*32kB 6*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 64584kB
HighMem: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 428kB
678229 total pagecache pages
14 pages in swap cache
Swap cache stats: add 11572, delete 11558, find 2617/3967
Free swap  = 30713236kB
Total swap = 30716152kB
720352 pages RAM
490976 pages HighMem
6570 pages reserved
1948 pages shared
692583 pages non-shared
Out of memory: kill process 3273 (sshconnect.sh) score 64 or a child
Killed process 3280 (ssh)
ssh invoked oom-killer: gfp_mask=0x1280d2, order=0, oomkilladj=0
Pid: 3327, comm: ssh Not tainted 2.6.27.54 #31
 [<c014ecd2>] oom_kill_process+0xe6/0x10c
 [<c014efd3>] out_of_memory+0xaf/0x100
 [<c0150844>] __alloc_pages_internal+0x31c/0x3b0
 [<c015b0ff>] do_anonymous_page+0x4f/0x128
 [<c015b773>] handle_mm_fault+0x19f/0x1e0
 [<c0113e07>] do_page_fault+0x11f/0x43c
 [<c0113ce8>] ? do_page_fault+0x0/0x43c
 [<c026f572>] error_code+0x72/0x78
 [<c01cb286>] ? __copy_to_user_ll+0x46/0x178
 [<c01780af>] ? __follow_mount+0xb/0x70
 [<c014c1cf>] file_read_actor+0x47/0x148
 [<c0153a3f>] ? activate_page+0x97/0xac
 [<c014bf2d>] do_generic_file_read+0x1a5/0x400
 [<c014c3fe>] generic_file_aio_read+0x9e/0x1b0
 [<c014c188>] ? file_read_actor+0x0/0x148
 [<c0170e38>] do_sync_read+0xc8/0x118
 [<c015d6b2>] ? mmap_region+0x30e/0x424
 [<c01328bc>] ? autoremove_wake_function+0x0/0x38
 [<c015da16>] ? arch_get_unmapped_area_topdown+0x136/0x158
 [<c0170ca2>] ? rw_verify_area+0xe/0xa0
 [<c0170f10>] vfs_read+0x88/0x124
 [<c0170d70>] ? do_sync_read+0x0/0x118
 [<c01720ac>] ? fget_light+0xc/0xa4
 [<c0171228>] sys_read+0x40/0x68
 [<c0103526>] syscall_call+0x7/0xb
 =======================
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd: 172
CPU    1: hi:  186, btch:  31 usd: 153
CPU    2: hi:  186, btch:  31 usd: 162
CPU    3: hi:  186, btch:  31 usd: 178
HighMem per-cpu:
CPU    0: hi:  186, btch:  31 usd:  80
CPU    1: hi:  186, btch:  31 usd: 157
CPU    2: hi:  186, btch:  31 usd: 208
CPU    3: hi:  186, btch:  31 usd: 137
Active:358103 inactive:320166 dirty:0 writeback:0 unstable:0
 free:19054 slab:7073 mapped:537 pagetables:77 bounce:0
DMA free:11164kB min:64kB low:80kB high:96kB active:1472kB inactive:1240kB present:15868kB pages_scanned:11371 all_unreclaimable? yes
lowmem_reserve[]: 0 873 2776 2776
Normal free:64624kB min:3744kB low:4680kB high:5616kB active:382440kB inactive:383044kB present:894080kB pages_scanned:903678 all_unreclaimable? no
lowmem_reserve[]: 0 0 15223 15223
HighMem free:428kB min:512kB low:2552kB high:4592kB active:1048500kB inactive:896492kB present:1948560kB pages_scanned:31715649 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 7*8kB 40*16kB 75*32kB 36*64kB 9*128kB 4*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 11164kB
Normal: 15316*4kB 22*8kB 11*16kB 2*32kB 6*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 64624kB
HighMem: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 428kB
678265 total pagecache pages
61 pages in swap cache
Swap cache stats: add 12299, delete 12238, find 2815/4300
Free swap  = 30713708kB
Total swap = 30716152kB
720352 pages RAM
490976 pages HighMem
6570 pages reserved
1505 pages shared
692612 pages non-shared
Out of memory: kill process 3325 (sshconnect.sh) score 64 or a child
Killed process 3327 (ssh)
sshconnect.sh invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0
Pid: 3325, comm: sshconnect.sh Not tainted 2.6.27.54 #31
 [<c014ecd2>] oom_kill_process+0xe6/0x10c
 [<c014efd3>] out_of_memory+0xaf/0x100
 [<c0150844>] __alloc_pages_internal+0x31c/0x3b0
 [<c0162251>] read_swap_cache_async+0x81/0xb4
 [<c01622d0>] swapin_readahead+0x4c/0x84
 [<c015afc9>] do_swap_page+0x16d/0x254
 [<c015b73c>] handle_mm_fault+0x168/0x1e0
 [<c0113e07>] do_page_fault+0x11f/0x43c
 [<c0113ce8>] ? do_page_fault+0x0/0x43c
 [<c026f572>] error_code+0x72/0x78
 =======================
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd: 170
CPU    1: hi:  186, btch:  31 usd: 160
CPU    2: hi:  186, btch:  31 usd: 162
CPU    3: hi:  186, btch:  31 usd: 178
HighMem per-cpu:
CPU    0: hi:  186, btch:  31 usd:  80
CPU    1: hi:  186, btch:  31 usd: 157
CPU    2: hi:  186, btch:  31 usd: 208
CPU    3: hi:  186, btch:  31 usd: 137
Active:358117 inactive:320156 dirty:0 writeback:0 unstable:0
 free:19108 slab:7072 mapped:336 pagetables:69 bounce:0
DMA free:11164kB min:64kB low:80kB high:96kB active:1472kB inactive:1240kB present:15868kB pages_scanned:11371 all_unreclaimable? yes
lowmem_reserve[]: 0 873 2776 2776
Normal free:64840kB min:3744kB low:4680kB high:5616kB active:382444kB inactive:382888kB present:894080kB pages_scanned:191395 all_unreclaimable? no
lowmem_reserve[]: 0 0 15223 15223
HighMem free:428kB min:512kB low:2552kB high:4592kB active:1048552kB inactive:896496kB present:1948560kB pages_scanned:31719106 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 7*8kB 40*16kB 75*32kB 36*64kB 9*128kB 4*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 11164kB
Normal: 15370*4kB 27*8kB 12*16kB 2*32kB 6*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 64896kB
HighMem: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 428kB
678251 total pagecache pages
38 pages in swap cache
Swap cache stats: add 12434, delete 12396, find 2851/4356
Free swap  = 30714276kB
Total swap = 30716152kB
720352 pages RAM
490976 pages HighMem
6570 pages reserved
849 pages shared
692792 pages non-shared
Out of memory: kill process 3317 (cronscript.sh) score 56 or a child
Killed process 3325 (sshconnect.sh)
-bash-3.1# exit
logout

============
Solution use "Allocate 3rd-level pagetables from highmem enabled in kernel"
===============

Note that LowFree is almost 845MB compared to about 74MB without this option enabled.
I believe I am correct about this kernel option, it should be enabled by default or strongly advised not to be disabled.


MemTotal:      2855128 kB
MemFree:       2402716 kB
Buffers:           496 kB
Cached:         129748 kB
SwapCached:          0 kB
Active:         295048 kB
Inactive:       108476 kB
HighTotal:     1963904 kB
HighFree:      1557316 kB
LowTotal:       891224 kB
LowFree:        845400 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:               0 kB
Writeback:           0 kB
AnonPages:      273288 kB
Mapped:           3380 kB
Slab:            26836 kB
SReclaimable:    21348 kB
SUnreclaim:       5488 kB
PageTables:        692 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
WritebackTmp:        0 kB
CommitLimit:   1427564 kB
Committed_AS:   284300 kB
VmallocTotal:   114680 kB
VmallocUsed:      1592 kB
VmallocChunk:   112784 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     4096 kB
DirectMap4k:      4096 kB
DirectMap4M:    913408 kB
 

It's not all about the numbers either, this kernel option stopped my kernel from triggering the OOM killer.


Tags:

linux, oom, sshd, mb, gigs, ram, kernel, dd, ve, paging, enabled, swapspace, livecd, swap, proc, swaps, filename, dev, sda, partition, http, www, redhat, archives, august, msg, html, quot, processor, features, resolve, allocate, rd, pagetables, highmem, meminfo, active, bash, memtotal, kb, memfree, buffers, cached, swapcached, inactive, hightotal, highfree, lowtotal, lowfree, swaptotal, swapfree, writeback, anonpages, mapped, slab, sreclaimable, sunreclaim, nfs_unstable, bounce, writebacktmp, commitlimit, committed_as, vmalloctotal, vmallocused, vmallocchunk, hugepages_total, hugepages_free, hugepages_rsvd, hugepages_surp, hugepagesize, directmap, dmesg, reserved, invoked, gfp_mask, oomkilladj, pid, comm, tainted, ecd, oom_kill_process, xe, efd, out_of_memory, xaf, __alloc_pages_internal, do_wp_page, task_new_fair, xb, handle_mm_fault, do_page_fault, ce, error_code, mem, info, dma, cpu, btch, usd, unstable, min, pages_scanned, all_unreclaimable, lowmem_reserve, pagecache, cache, stats, delete, cronscript, sh, __wake_up_bit, xc, read_swap_cache_async, swapin_readahead, afc, do_swap_page, cb, __copy_to_user_ll, setup_frame, dequeue_signal, ptrace_signal, xa, xbc, handle_signal, do_signal, af, sys_select, do_notify_resume, work_notifysig, sshconnect, ssh, ff, do_anonymous_page, __follow_mount, cf, file_read_actor, activate_page, xac, bf, do_generic_file_read, fe, generic_file_aio_read, do_sync_read, mmap_region, bc, autoremove_wake_function, arch_get_unmapped_area_topdown, ca, rw_verify_area, vfs_read, ac, fget_light, sys_read, syscall_call, logout, default, advised, disabled, triggering,

Latest Articles

  • How to allow SSH root user access in Linux/Debian/Mint/RHEL/Ubuntu/CentOS
  • Ansible Tutorial - Playbook How To Install From Scratch and Deploy LAMP + Wordpress on Remote Server
  • Ceph Install Errors on Proxmox / How To Fix Solution
  • Proxmox Update Error https://enterprise.proxmox.com/debian/pve bullseye InRelease 401 Unauthorized [IP: 144.217.225.162 443]
  • QEMU/KVM How to Hot-add A Virtual Disk .raw/.qcow2 via QEMU Monitor Commands
  • Proxmox How To Enable Ceph Distributed Storage Cluster with OSD and Pools
  • pulseaudio issue on QEMU/KVM guest VM when microphone is replugged/unplugged pulseaudio: pa_threaded_mainloop_lock failed pulseaudio: Reason: Invalid argument
  • Ubuntu Linux Mint - Volume Control Stopped Working
  • Proxmox Services Won't Start Failed to start The Proxmox VE cluster filesystem. Proxmox VE firewall. PVE Status Daemon. Proxmox VE scheduler. PVE Cluster HA Resource Manager Daemon. PVE Local HA Resource Manager Daemon.
  • Proxmox Guide FAQ / Errors / Howto
  • Virtualbox Vbox Issue Cannot Enable Nested Virtualization Button is Grayed/Greyed Out and Unclickable HowTo Solution
  • Virtualbox VBOX Howto Port Forward To Guests
  • Linux Ubuntu Debian Centos Mint - How To Check if Intel VT-x or AMD-V Hardware Virtualization is Enabled?
  • Linux Howto Zip Multiple Files and Directories
  • Windows Cannot Format USB drive Device Media is Write Protected Error Solution
  • Linux Mint 20 cannot install snapd missing solution
  • Virtualbox VBOX How To Install Guest-Utils/GuestUtils so drag and drop and clipboard works Ubuntu Mint Debian Linux
  • How to install Kubernetes with microk8s and deploy apps on Debian/Mint/Ubuntu Linux
  • vi how to delete everything to the end of the line or the rest of the line from the cursor
  • Cisco Howto Configure Console Port/Terminal/Comm Server with Async Cable Setup