Linux Out of Memory OOM Object Killer Solution "Out of memory: kill process 1955 (sshd) score 81 or a child"

I had a system running a 128MB live CD image with 2.8 gigs of available RAM and the OOM kernel killer went crazy when using dd for more than 8 minutes and kept killing everything. I've read that this is due to a low-memory issue and paging in the kernel and 32-bit systems with lots of RAM.

I even enabled swapspace on my LiveCD and the issue happened 25 minutes into dd rather than 8 minutes, so what gives?
Also no swap space was ever used!

cat /proc/swaps
Filename                Type        Size    Used    Priority
/dev/sda2                               partition    30716152    2784    -1

Here is a nice thread on the issue: http://www.redhat.com/archives/redhat-list/2007-August/msg00060.html
I believe the option below in the kernel under "Processor Type & Features" should resolve this.

Solution for OOM

[*] Allocate 3rd-level pagetables from highmem

Here is meminfo when the OOM killer is not active

-bash-3.1# cat /proc/meminfo
MemTotal:      2855128 kB
MemFree:         77016 kB
Buffers:         47012 kB
Cached:        2646720 kB
SwapCached:        908 kB
Active:        1411204 kB
Inactive:      1283876 kB
HighTotal:     1963904 kB
HighFree:          428 kB
LowTotal:       891224 kB
LowFree:         76588 kB
SwapTotal:    30716152 kB
SwapFree:     30712628 kB
Dirty:               0 kB
Writeback:         296 kB
AnonPages:         876 kB
Mapped:           1424 kB
Slab:            45668 kB
SReclaimable:    40420 kB
SUnreclaim:       5248 kB
PageTables:        328 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
WritebackTmp:        0 kB
CommitLimit: 32143716 kB
Committed_AS:     8416 kB
VmallocTotal:   114680 kB
VmallocUsed:     16336 kB
VmallocChunk:    97976 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     4096 kB
DirectMap4k:      4096 kB
DirectMap4M:    913408 kB

Here is meminfo when OOM killer is active

cat /proc/meminfo
MemTotal:      2855128 kB
MemFree:         76440 kB
Buffers:             0 kB
Cached:        2712884 kB
SwapCached:        520 kB
Active:        1433708 kB
Inactive:      1279804 kB
HighTotal:     1963904 kB
HighFree:          428 kB
LowTotal:       891224 kB
LowFree:         76012 kB
SwapTotal:    30716152 kB
SwapFree:     30713364 kB
Dirty:               0 kB
Writeback:           0 kB
AnonPages:         372 kB
Mapped:           2616 kB
Slab:            28280 kB
SReclaimable:    22952 kB
SUnreclaim:       5328 kB
PageTables:        320 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
WritebackTmp:        0 kB
CommitLimit: 32143716 kB
Committed_AS:     6888 kB
VmallocTotal:   114680 kB
VmallocUsed:     16336 kB
VmallocChunk:    97976 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     4096 kB
DirectMap4k:      4096 kB
DirectMap4M:    913408 kB

dmesg

2kB
720352 pages RAM
490976 pages HighMem
6570 pages reserved
1585 pages shared
692675 pages non-shared
Out of memory: kill process 1955 (sshd) score 81 or a child
Killed process 3302 (sshd)
bash invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0
Pid: 3311, comm: bash Not tainted 2.6.27.54 #31
[<c014ecd2>] oom_kill_process+0xe6/0x10c
[<c014efd3>] out_of_memory+0xaf/0x100
[<c0150844>] __alloc_pages_internal+0x31c/0x3b0
[<c015a53f>] do_wp_page+0x9f/0x49c
[<c011855e>] ? task_new_fair+0x6e/0xb0
[<c015b757>] handle_mm_fault+0x183/0x1e0
[<c0113e07>] do_page_fault+0x11f/0x43c
[<c0113ce8>] ? do_page_fault+0x0/0x43c
[<c026f572>] error_code+0x72/0x78
=======================
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi: 186, btch: 31 usd: 150
CPU    1: hi: 186, btch: 31 usd: 171
CPU    2: hi: 186, btch: 31 usd: 145
CPU    3: hi: 186, btch: 31 usd: 153
HighMem per-cpu:
CPU    0: hi: 186, btch: 31 usd: 80
CPU    1: hi: 186, btch: 31 usd: 157
CPU    2: hi: 186, btch: 31 usd: 208
CPU    3: hi: 186, btch: 31 usd: 137
Active:358187 inactive:320084 dirty:0 writeback:0 unstable:0
free:19121 slab:7075 mapped:659 pagetables:82 bounce:0
DMA free:11164kB min:64kB low:80kB high:96kB active:1472kB inactive:1240kB present:15868kB pages_scanned:9835 all_unreclaimable? yes
lowmem_reserve[]: 0 873 2776 2776
Normal free:64892kB min:3744kB low:4680kB high:5616kB active:382712kB inactive:382612kB present:894080kB pages_scanned:372896 all_unreclaimable? no
lowmem_reserve[]: 0 0 15223 15223
HighMem free:428kB min:512kB low:2552kB high:4592kB active:1048564kB inactive:896484kB present:1948560kB pages_scanned:31617081 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 7*8kB 40*16kB 75*32kB 36*64kB 9*128kB 4*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 11164kB
Normal: 15423*4kB 18*8kB 9*16kB 2*32kB 6*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 64988kB
HighMem: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 428kB
678222 total pagecache pages
28 pages in swap cache
Swap cache stats: add 9762, delete 9734, find 2155/3264
Free swap = 30713264kB
Total swap = 30716152kB
720352 pages RAM
490976 pages HighMem
6570 pages reserved
1911 pages shared
692514 pages non-shared
Out of memory: kill process 1955 (sshd) score 81 or a child
Killed process 3310 (sshd)
cronscript.sh invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0
Pid: 3322, comm: cronscript.sh Not tainted 2.6.27.54 #31
[<c014ecd2>] oom_kill_process+0xe6/0x10c
[<c014efd3>] out_of_memory+0xaf/0x100
[<c0150844>] __alloc_pages_internal+0x31c/0x3b0
[<c015a53f>] do_wp_page+0x9f/0x49c
[<c0132948>] ? __wake_up_bit+0xc/0x40
[<c015b757>] handle_mm_fault+0x183/0x1e0
[<c0113e07>] do_page_fault+0x11f/0x43c
[<c0113ce8>] ? do_page_fault+0x0/0x43c
[<c026f572>] error_code+0x72/0x78
=======================
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi: 186, btch: 31 usd: 143
CPU    1: hi: 186, btch: 31 usd: 161
CPU    2: hi: 186, btch: 31 usd: 142
CPU    3: hi: 186, btch: 31 usd: 171
HighMem per-cpu:
CPU    0: hi: 186, btch: 31 usd: 80
CPU    1: hi: 186, btch: 31 usd: 157
CPU    2: hi: 186, btch: 31 usd: 208
CPU    3: hi: 186, btch: 31 usd: 137
Active:358354 inactive:319939 dirty:0 writeback:0 unstable:0
free:19071 slab:7075 mapped:689 pagetables:111 bounce:0
DMA free:11164kB min:64kB low:80kB high:96kB active:1472kB inactive:1240kB present:15868kB pages_scanned:10539 all_unreclaimable? yes
lowmem_reserve[]: 0 873 2776 2776
Normal free:64692kB min:3744kB low:4680kB high:5616kB active:383072kB inactive:382340kB present:894080kB pages_scanned:1310990 all_unreclaimable? yes
lowmem_reserve[]: 0 0 15223 15223
HighMem free:428kB min:512kB low:2552kB high:4592kB active:1048872kB inactive:896176kB present:1948560kB pages_scanned:31666018 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 7*8kB 40*16kB 75*32kB 36*64kB 9*128kB 4*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 11164kB
Normal: 15343*4kB 21*8kB 9*16kB 2*32kB 6*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 64692kB
HighMem: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 428kB
678265 total pagecache pages
61 pages in swap cache
Swap cache stats: add 11379, delete 11318, find 2543/3850
Free swap = 30712744kB
Total swap = 30716152kB
720352 pages RAM
490976 pages HighMem
6570 pages reserved
2492 pages shared
692547 pages non-shared
Out of memory: kill process 3318 (cronscript.sh) score 107 or a child
Killed process 3319 (cronscript.sh)
bash invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0
Pid: 3323, comm: bash Not tainted 2.6.27.54 #31
[<c014ecd2>] oom_kill_process+0xe6/0x10c
[<c014efd3>] out_of_memory+0xaf/0x100
[<c0150844>] __alloc_pages_internal+0x31c/0x3b0
[<c015a53f>] do_wp_page+0x9f/0x49c
[<c015b757>] handle_mm_fault+0x183/0x1e0
[<c0113e07>] do_page_fault+0x11f/0x43c
[<c0113ce8>] ? do_page_fault+0x0/0x43c
[<c026f572>] error_code+0x72/0x78
=======================
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi: 186, btch: 31 usd: 178
CPU    1: hi: 186, btch: 31 usd: 180
CPU    2: hi: 186, btch: 31 usd: 171
CPU    3: hi: 186, btch: 31 usd: 179
HighMem per-cpu:
CPU    0: hi: 186, btch: 31 usd: 80
CPU    1: hi: 186, btch: 31 usd: 157
CPU    2: hi: 186, btch: 31 usd: 208
CPU    3: hi: 186, btch: 31 usd: 137
Active:358137 inactive:320124 dirty:0 writeback:0 unstable:0
free:19044 slab:7075 mapped:659 pagetables:100 bounce:0
DMA free:11164kB min:64kB low:80kB high:96kB active:1464kB inactive:1240kB present:15868kB pages_scanned:10731 all_unreclaimable? yes
lowmem_reserve[]: 0 873 2776 2776
Normal free:64584kB min:3744kB low:4680kB high:5616kB active:382404kB inactive:382720kB present:894080kB pages_scanned:1952385 all_unreclaimable? no
lowmem_reserve[]: 0 0 15223 15223
HighMem free:428kB min:512kB low:2552kB high:4592kB active:1048680kB inactive:896424kB present:1948560kB pages_scanned:31674398 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 7*8kB 40*16kB 75*32kB 36*64kB 9*128kB 4*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 11164kB
Normal: 15332*4kB 13*8kB 9*16kB 2*32kB 6*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 64584kB
HighMem: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 428kB
678229 total pagecache pages
14 pages in swap cache
Swap cache stats: add 11572, delete 11558, find 2617/3965
Free swap = 30712912kB
Total swap = 30716152kB
720352 pages RAM
490976 pages HighMem
6570 pages reserved
2337 pages shared
692510 pages non-shared
Out of memory: kill process 1955 (sshd) score 81 or a child
Killed process 3313 (sshd)
sshd invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0
Pid: 1955, comm: sshd Not tainted 2.6.27.54 #31
[<c014ecd2>] oom_kill_process+0xe6/0x10c
[<c014efd3>] out_of_memory+0xaf/0x100
[<c0150844>] __alloc_pages_internal+0x31c/0x3b0
[<c0162251>] read_swap_cache_async+0x81/0xb4
[<c01622d0>] swapin_readahead+0x4c/0x84
[<c015afc9>] do_swap_page+0x16d/0x254
[<c015b73c>] handle_mm_fault+0x168/0x1e0
[<c0113e07>] do_page_fault+0x11f/0x43c
[<c0113ce8>] ? do_page_fault+0x0/0x43c
[<c026f572>] error_code+0x72/0x78
[<c01cb286>] ? __copy_to_user_ll+0x46/0x178
[<c0102a9f>] setup_frame+0x10b/0x324
[<c0129e7d>] ? dequeue_signal+0x29/0x114
[<c012b4a6>] ? ptrace_signal+0xa/0xbc
[<c0103266>] handle_signal+0x116/0x15c
[<c0103322>] do_signal+0x76/0x128
[<c015af66>] ? do_swap_page+0x10a/0x254
[<c017d4f6>] ? sys_select+0x9e/0x15c
[<c01033f5>] do_notify_resume+0x21/0x24
[<c01035b6>] work_notifysig+0x13/0x19
=======================
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi: 186, btch: 31 usd: 183
CPU    1: hi: 186, btch: 31 usd: 180
CPU    2: hi: 186, btch: 31 usd: 171
CPU    3: hi: 186, btch: 31 usd: 179
HighMem per-cpu:
CPU    0: hi: 186, btch: 31 usd: 80
CPU    1: hi: 186, btch: 31 usd: 157
CPU    2: hi: 186, btch: 31 usd: 208
CPU    3: hi: 186, btch: 31 usd: 137
Active:358256 inactive:320033 dirty:0 writeback:0 unstable:0
free:19044 slab:7075 mapped:613 pagetables:100 bounce:0
DMA free:11164kB min:64kB low:80kB high:96kB active:1464kB inactive:1240kB present:15868kB pages_scanned:10731 all_unreclaimable? yes
lowmem_reserve[]: 0 873 2776 2776
Normal free:64584kB min:3744kB low:4680kB high:5616kB active:382836kB inactive:382456kB present:894080kB pages_scanned:1975428 all_unreclaimable? yes
lowmem_reserve[]: 0 0 15223 15223
HighMem free:428kB min:512kB low:2552kB high:4592kB active:1048724kB inactive:896436kB present:1948560kB pages_scanned:31675298 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 7*8kB 40*16kB 75*32kB 36*64kB 9*128kB 4*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 11164kB
Normal: 15332*4kB 13*8kB 9*16kB 2*32kB 6*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 64584kB
HighMem: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 428kB
678229 total pagecache pages
14 pages in swap cache
Swap cache stats: add 11572, delete 11558, find 2617/3967
Free swap = 30713236kB
Total swap = 30716152kB
720352 pages RAM
490976 pages HighMem
6570 pages reserved
1948 pages shared
692583 pages non-shared
Out of memory: kill process 3273 (sshconnect.sh) score 64 or a child
Killed process 3280 (ssh)
ssh invoked oom-killer: gfp_mask=0x1280d2, order=0, oomkilladj=0
Pid: 3327, comm: ssh Not tainted 2.6.27.54 #31
[<c014ecd2>] oom_kill_process+0xe6/0x10c
[<c014efd3>] out_of_memory+0xaf/0x100
[<c0150844>] __alloc_pages_internal+0x31c/0x3b0
[<c015b0ff>] do_anonymous_page+0x4f/0x128
[<c015b773>] handle_mm_fault+0x19f/0x1e0
[<c0113e07>] do_page_fault+0x11f/0x43c
[<c0113ce8>] ? do_page_fault+0x0/0x43c
[<c026f572>] error_code+0x72/0x78
[<c01cb286>] ? __copy_to_user_ll+0x46/0x178
[<c01780af>] ? __follow_mount+0xb/0x70
[<c014c1cf>] file_read_actor+0x47/0x148
[<c0153a3f>] ? activate_page+0x97/0xac
[<c014bf2d>] do_generic_file_read+0x1a5/0x400
[<c014c3fe>] generic_file_aio_read+0x9e/0x1b0
[<c014c188>] ? file_read_actor+0x0/0x148
[<c0170e38>] do_sync_read+0xc8/0x118
[<c015d6b2>] ? mmap_region+0x30e/0x424
[<c01328bc>] ? autoremove_wake_function+0x0/0x38
[<c015da16>] ? arch_get_unmapped_area_topdown+0x136/0x158
[<c0170ca2>] ? rw_verify_area+0xe/0xa0
[<c0170f10>] vfs_read+0x88/0x124
[<c0170d70>] ? do_sync_read+0x0/0x118
[<c01720ac>] ? fget_light+0xc/0xa4
[<c0171228>] sys_read+0x40/0x68
[<c0103526>] syscall_call+0x7/0xb
=======================
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi: 186, btch: 31 usd: 172
CPU    1: hi: 186, btch: 31 usd: 153
CPU    2: hi: 186, btch: 31 usd: 162
CPU    3: hi: 186, btch: 31 usd: 178
HighMem per-cpu:
CPU    0: hi: 186, btch: 31 usd: 80
CPU    1: hi: 186, btch: 31 usd: 157
CPU    2: hi: 186, btch: 31 usd: 208
CPU    3: hi: 186, btch: 31 usd: 137
Active:358103 inactive:320166 dirty:0 writeback:0 unstable:0
free:19054 slab:7073 mapped:537 pagetables:77 bounce:0
DMA free:11164kB min:64kB low:80kB high:96kB active:1472kB inactive:1240kB present:15868kB pages_scanned:11371 all_unreclaimable? yes
lowmem_reserve[]: 0 873 2776 2776
Normal free:64624kB min:3744kB low:4680kB high:5616kB active:382440kB inactive:383044kB present:894080kB pages_scanned:903678 all_unreclaimable? no
lowmem_reserve[]: 0 0 15223 15223
HighMem free:428kB min:512kB low:2552kB high:4592kB active:1048500kB inactive:896492kB present:1948560kB pages_scanned:31715649 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 7*8kB 40*16kB 75*32kB 36*64kB 9*128kB 4*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 11164kB
Normal: 15316*4kB 22*8kB 11*16kB 2*32kB 6*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 64624kB
HighMem: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 428kB
678265 total pagecache pages
61 pages in swap cache
Swap cache stats: add 12299, delete 12238, find 2815/4300
Free swap = 30713708kB
Total swap = 30716152kB
720352 pages RAM
490976 pages HighMem
6570 pages reserved
1505 pages shared
692612 pages non-shared
Out of memory: kill process 3325 (sshconnect.sh) score 64 or a child
Killed process 3327 (ssh)
sshconnect.sh invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0
Pid: 3325, comm: sshconnect.sh Not tainted 2.6.27.54 #31
[<c014ecd2>] oom_kill_process+0xe6/0x10c
[<c014efd3>] out_of_memory+0xaf/0x100
[<c0150844>] __alloc_pages_internal+0x31c/0x3b0
[<c0162251>] read_swap_cache_async+0x81/0xb4
[<c01622d0>] swapin_readahead+0x4c/0x84
[<c015afc9>] do_swap_page+0x16d/0x254
[<c015b73c>] handle_mm_fault+0x168/0x1e0
[<c0113e07>] do_page_fault+0x11f/0x43c
[<c0113ce8>] ? do_page_fault+0x0/0x43c
[<c026f572>] error_code+0x72/0x78
=======================
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi: 186, btch: 31 usd: 170
CPU    1: hi: 186, btch: 31 usd: 160
CPU    2: hi: 186, btch: 31 usd: 162
CPU    3: hi: 186, btch: 31 usd: 178
HighMem per-cpu:
CPU    0: hi: 186, btch: 31 usd: 80
CPU    1: hi: 186, btch: 31 usd: 157
CPU    2: hi: 186, btch: 31 usd: 208
CPU    3: hi: 186, btch: 31 usd: 137
Active:358117 inactive:320156 dirty:0 writeback:0 unstable:0
free:19108 slab:7072 mapped:336 pagetables:69 bounce:0
DMA free:11164kB min:64kB low:80kB high:96kB active:1472kB inactive:1240kB present:15868kB pages_scanned:11371 all_unreclaimable? yes
lowmem_reserve[]: 0 873 2776 2776
Normal free:64840kB min:3744kB low:4680kB high:5616kB active:382444kB inactive:382888kB present:894080kB pages_scanned:191395 all_unreclaimable? no
lowmem_reserve[]: 0 0 15223 15223
HighMem free:428kB min:512kB low:2552kB high:4592kB active:1048552kB inactive:896496kB present:1948560kB pages_scanned:31719106 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 7*8kB 40*16kB 75*32kB 36*64kB 9*128kB 4*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 11164kB
Normal: 15370*4kB 27*8kB 12*16kB 2*32kB 6*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 64896kB
HighMem: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 428kB
678251 total pagecache pages
38 pages in swap cache
Swap cache stats: add 12434, delete 12396, find 2851/4356
Free swap = 30714276kB
Total swap = 30716152kB
720352 pages RAM
490976 pages HighMem
6570 pages reserved
849 pages shared
692792 pages non-shared
Out of memory: kill process 3317 (cronscript.sh) score 56 or a child
Killed process 3325 (sshconnect.sh)
-bash-3.1# exit
logout

============
Solution use "Allocate 3rd-level pagetables from highmem enabled in kernel"
===============
Note that LowFree is almost 845MB compared to about 74MB without this option enabled.
I believe I am correct about this kernel option, it should be enabled by default or strongly advised not to be disabled.

MemTotal:      2855128 kB
MemFree:       2402716 kB
Buffers:           496 kB
Cached:         129748 kB
SwapCached:          0 kB
Active:         295048 kB
Inactive:       108476 kB
HighTotal:     1963904 kB
HighFree:      1557316 kB
LowTotal:       891224 kB
LowFree:        845400 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:               0 kB
Writeback:           0 kB
AnonPages:      273288 kB
Mapped:           3380 kB
Slab:            26836 kB
SReclaimable:    21348 kB
SUnreclaim:       5488 kB
PageTables:        692 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
WritebackTmp:        0 kB
CommitLimit:   1427564 kB
Committed_AS:   284300 kB
VmallocTotal:   114680 kB
VmallocUsed:      1592 kB
VmallocChunk:   112784 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     4096 kB
DirectMap4k:      4096 kB
DirectMap4M:    913408 kB