Openvz OOM Centos Issue Memory Config Settings Solution

Linux box13. 2.6.32-042stab076.5 #1 SMP Mon Mar 18 20:41:34 MSK 2013 x86_64 x86_64 x86_64 GNU/Linux


even setting privvmpages to a specific setting DOES not affect "free -m" in containers.
This is probably a kernel issue

 23:36:29 up 159 days,  7:12,  4 users,  load average: 0.42, 0.44, 0.33
[root@box13 ~]# free -m
             total       used       free     shared    buffers     cached
Mem:         15914      12620       3294          0       1314       6530
-/+ buffers/cache:       4774      11139
Swap:        29998        133      29865


[13722911.849381] Total 655262 anon:165965 file:489297 a_anon:34353 i_anon:131612 a_file:1556 i_file:487741 unevictable:0
[13722911.849407] RAM: 65536 / 65536 [8138] SWAP: 131072 / 131072 [8107] KMEM: 17920000 / inf [0] DCSZ: 3982704 / 2147483646 [0] OOMG: 155142 / inf [3681] Dirty 195 Wback 0 Dche 2872 Prnd 10618393
[13722911.849648] Out of memory in UB: Kill process 32324 (httpd) score -803 or sacrifice child
[13722911.858363] OOM killed process 32324 (httpd) vm:281004kB, rss:123992kB, swap:30436kB
[13722911.884909] OOM killed process httpd (pid=32324, ve=50200) exited, free=51056.
[13722911.884915] UB-50200-Mem-Info:
[13722911.884923] Node 0 DMA prio:41 portion:60 scan:0 a_anon:0 11591658ms i_anon:0 11591658ms a_file:0 11591658ms i_file:0 11591658ms unevictable:0 reclaim_stat: 0 0 0 0
[13722911.884931] Node 0 DMA/shadow prio:41 portion:0 scan:0 a_anon:0 2891651ms i_anon:0 2891651ms a_file:0 2891651ms i_file:0 2891651ms unevictable:0 reclaim_stat: 0 0 0 0
[13722911.884940] Node 0 DMA32 prio:14 portion:14245 scan:0 a_anon:0 11646ms i_anon:1 11646ms a_file:0 11646ms i_file:0 11646ms unevictable:0 reclaim_stat: 1126 910 0 0
[13722911.884949] Node 0 DMA32/shadow prio:41 portion:0 scan:0 a_anon:32 633962136ms i_anon:2390 633962136ms a_file:200 120071698ms i_file:27975 120851698ms unevictable:0 reclaim_stat: 89 41 2484 75
[13722911.884958] Node 0 Normal prio:14 portion:51230 scan:76 a_anon:22683 11646ms i_anon:15817 11646ms a_file:1 11646ms i_file:0 11646ms unevictable:0 reclaim_stat: 12407 2819 2 2
[13722911.884967] Node 0 Normal/shadow prio:41 portion:0 scan:0 a_anon:4025 159791964ms i_anon:91979 159791964ms a_file:1357 67751686ms i_file:459761 67871686ms unevictable:0 reclaim_stat: 7627 6311 32980 3681
[13722911.884976] Total 626221 anon:136927 file:489294 a_anon:26740 i_anon:110187 a_file:1558 i_file:487736 unevictable:0
[13722911.884981] RAM: 43602 / 65536 [8138] SWAP: 124166 / 131072 [8107] KMEM: 17379328 / inf [0] DCSZ: 3982704 / 2147483646 [0] OOMG: 155142 / inf [3682] Dirty 195 Wback 0 Dche 2872 Prnd 10618393
[13722911.884993] <<< 50200 oom generation 3681 ends (task died)
[13723466.860263] >>> 50200 oom generation 3682 starts
[13723466.860273] 34149 (httpd) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
[13723466.860278] UB-50200-Mem-Info:
[13723466.860285] Node 0 DMA prio:16 portion:60 scan:0 a_anon:0 1286627ms i_anon:0 1286627ms a_file:0 1286627ms i_file:0 1286627ms unevictable:0 reclaim_stat: 0 0 0 0
[13723466.860294] Node 0 DMA/shadow prio:41 portion:0 scan:0 a_anon:0 26601ms i_anon:0 26601ms a_file:0 26601ms i_file:0 26601ms unevictable:0 reclaim_stat: 0 0 0 0
[13723466.860304] Node 0 DMA32 prio:13 portion:14245 scan:0 a_anon:6131 26601ms i_anon:5814 26601ms a_file:0 26601ms i_file:2 26601ms unevictable:0 reclaim_stat: 2218 2205 2 0
[13723466.860314] Node 0 DMA32/shadow prio:41 portion:0 scan:0 a_anon:32 634517112ms i_anon:2862 634517112ms a_file:200 120626674ms i_file:28076 121406674ms unevictable:0 reclaim_stat: 89 41 2484 75
[13723466.860324] Node 0 Normal prio:13 portion:51230 scan:91 a_anon:24319 26601ms i_anon:24328 26601ms a_file:16 26601ms i_file:14 26601ms unevictable:0 reclaim_stat: 6160 2620 1383 542
[13723466.860335] Node 0 Normal/shadow prio:41 portion:0 scan:0 a_anon:4025 160346940ms i_anon:98256 160346940ms a_file:1442 68306662ms i_file:459397 68426662ms unevictable:0 reclaim_stat: 7627 6311 33449 3766
[13723466.860344] Total 654914 anon:165767 file:489147 a_anon:34507 i_anon:131260 a_file:1658 i_file:487489 unevictable:0
[13723466.860349] RAM: 65536 / 65536 [8139] SWAP: 131072 / 131072 [8108] KMEM: 18399232 / inf [0] DCSZ: 4026164 / 2147483646 [0] OOMG: 155142 / inf [3682] Dirty 130 Wback 0 Dche 2898 Prnd 10618626
[13723466.860535] Out of memory in UB: Kill process 32116 (httpd) score -887 or sacrifice child
[13723466.869214] OOM killed process 32116 (httpd) vm:214460kB, rss:82508kB, swap:6104kB
[13723466.879562] OOM killed process httpd (pid=32116, ve=50200) exited, free=35907.
[13723466.879570] UB-50200-Mem-Info:
[13723466.879577] Node 0 DMA prio:16 portion:60 scan:0 a_anon:0 1286646ms i_anon:0 1286646ms a_file:0 1286646ms i_file:0 1286646ms unevictable:0 reclaim_stat: 0 0 0 0
[13723466.879585] Node 0 DMA/shadow prio:41 portion:0 scan:0 a_anon:0 26620ms i_anon:0 26620ms a_file:0 26620ms i_file:0 26620ms unevictable:0 reclaim_stat: 0 0 0 0
[13723466.879594] Node 0 DMA32 prio:13 portion:14245 scan:0 a_anon:6131 26620ms i_anon:5814 26620ms a_file:0 26620ms i_file:2 26620ms unevictable:0 reclaim_stat: 2218 2205 2 0
[13723466.879603] Node 0 DMA32/shadow prio:41 portion:0 scan:0 a_anon:32 634517131ms i_anon:2861 634517131ms a_file:200 120626693ms i_file:28076 121406693ms unevictable:0 reclaim_stat: 89 41 2484 75
[13723466.879613] Node 0 Normal prio:13 portion:51230 scan:91 a_anon:24319 26620ms i_anon:12521 26620ms a_file:16 26620ms i_file:33 26620ms unevictable:0 reclaim_stat: 6175 2620 1402 542
[13723466.879625] Node 0 Normal/shadow prio:41 portion:0 scan:0 a_anon:4025 160346959ms i_anon:97456 160346959ms a_file:1443 68306681ms i_file:459385 68426681ms unevictable:0 reclaim_stat: 7627 6311 33450 3767
[13723466.879634] Total 642313 anon:153158 file:489155 a_anon:34507 i_anon:118651 a_file:1659 i_file:487496 unevictable:0
[13723466.879639] RAM: 54408 / 65536 [8139] SWAP: 129916 / 131072 [8108] KMEM: 17915904 / inf [0] DCSZ: 4026164 / 2147483646 [0] OOMG: 155142 / inf [3683] Dirty 130 Wback 0 Dche 2898 Prnd 10618626
[13723466.879652] <<< 50200 oom generation 3682 ends (task died)
[13726707.794190] >>> 50200 oom generation 3683 starts
[13726707.794200] 42508 (httpd) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
[13726707.794205] UB-50200-Mem-Info:
[13726707.794212] Node 0 DMA prio:24 portion:60 scan:0 a_anon:0 4527561ms i_anon:0 4527561ms a_file:0 4527561ms i_file:0 4527561ms unevictable:0 reclaim_stat: 0 0 0 0
[13726707.794220] Node 0 DMA/shadow prio:41 portion:0 scan:0 a_anon:0 27535ms i_anon:0 27535ms a_file:0 27535ms i_file:0 27535ms unevictable:0 reclaim_stat: 0 0 0 0
[13726707.794228] Node 0 DMA32 prio:13 portion:14245 scan:163 a_anon:315 27535ms i_anon:281 27535ms a_file:0 27535ms i_file:0 27535ms unevictable:0 reclaim_stat: 182 160 0 0
[13726707.794237] Node 0 DMA32/shadow prio:41 portion:0 scan:0 a_anon:32 637758046ms i_anon:12518 637758046ms a_file:200 123867608ms i_file:28705 124647608ms unevictable:0 reclaim_stat: 89 41 2493 84
[13726707.794246] Node 0 Normal prio:13 portion:51230 scan:273 a_anon:30105 27535ms i_anon:30150 27535ms a_file:20 27535ms i_file:0 27535ms unevictable:0 reclaim_stat: 11778 5001 80 30
[13726707.794288] Node 0 Normal/shadow prio:41 portion:0 scan:0 a_anon:4025 163587874ms i_anon:88812 163587874ms a_file:1498 71547596ms i_file:459644 71667596ms unevictable:0 reclaim_stat: 7627 6311 34529 3822
[13726707.794304] Total 656305 anon:166238 file:490067 a_anon:34477 i_anon:131761 a_file:1718 i_file:488349 unevictable:0
[13726707.794314] RAM: 65536 / 65536 [8140] SWAP: 131072 / 131072 [8109] KMEM: 18591744 / inf [0] DCSZ: 4026145 / 2147483646 [0] OOMG: 155142 / inf [3683] Dirty 130 Wback 0 Dche 2897 Prnd 10620313
[13726707.794527] Out of memory in UB: Kill process 42508 (httpd) score -874 or sacrifice child
[13726707.803145] OOM killed process 42508 (httpd) vm:224932kB, rss:89192kB, swap:10060kB
[13726707.823161] OOM killed process httpd (pid=42508, ve=50200) exited, free=39614.
[13726707.823167] UB-50200-Mem-Info:
[13726707.823174] Node 0 DMA prio:24 portion:60 scan:0 a_anon:0 4527589ms i_anon:0 4527589ms a_file:0 4527589ms i_file:0 4527589ms unevictable:0 reclaim_stat: 0 0 0 0
[13726707.823182] Node 0 DMA/shadow prio:41 portion:0 scan:0 a_anon:0 27563ms i_anon:0 27563ms a_file:0 27563ms i_file:0 27563ms unevictable:0 reclaim_stat: 0 0 0 0
[13726707.823190] Node 0 DMA32 prio:13 portion:14245 scan:0 a_anon:27 27563ms i_anon:0 27563ms a_file:0 27563ms i_file:0 27563ms unevictable:0 reclaim_stat: 157 112 2 1
[13726707.823199] Node 0 DMA32/shadow prio:41 portion:0 scan:0 a_anon:32 637758074ms i_anon:10975 637758074ms a_file:200 123867636ms i_file:28707 124647636ms unevictable:0 reclaim_stat: 89 41 2493 84
[13726707.823208] Node 0 Normal prio:13 portion:51230 scan:0 a_anon:29591 27563ms i_anon:17907 27563ms a_file:0 27563ms i_file:0 27563ms unevictable:0 reclaim_stat: 13594 6683 80 16
[13726707.823218] Node 0 Normal/shadow prio:41 portion:0 scan:0 a_anon:4025 163587902ms i_anon:88583 163587902ms a_file:1577 71547624ms i_file:459588 71667624ms unevictable:0 reclaim_stat: 7627 6311 34608 3901
[13726707.823227] Total 641212 anon:151140 file:490072 a_anon:33675 i_anon:117465 a_file:1777 i_file:488295 unevictable:0
[13726707.823232] RAM: 52362 / 65536 [8140] SWAP: 129048 / 131072 [8109] KMEM: 18141184 / inf [0] DCSZ: 4026145 / 2147483646 [0] OOMG: 155142 / inf [3684] Dirty 130 Wback 0 Dche 2897 Prnd 10620313
[13726707.823244] <<< 50200 oom generation 3683 ends (task died)
[13732772.685609] >>> 50200 oom generation 3684 starts
[13732772.685619] 67753 (httpd) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
[13732772.685623] UB-50200-Mem-Info:
[13732772.685629] Node 0 DMA prio:40 portion:60 scan:0 a_anon:0 10592452ms i_anon:0 10592452ms a_file:0 10592452ms i_file:0 10592452ms unevictable:0 reclaim_stat: 0 0 0 0
[13732772.685638] Node 0 DMA/shadow prio:41 portion:0 scan:0 a_anon:0 1952418ms i_anon:0 1952418ms a_file:0 1952418ms i_file:0 1952418ms unevictable:0 reclaim_stat: 0 0 0 0
[13732772.685646] Node 0 DMA32 prio:13 portion:14245 scan:0 a_anon:3982 32418ms i_anon:3990 32418ms a_file:0 32418ms i_file:0 32418ms unevictable:0 reclaim_stat: 1386 859 0 0
[13732772.685655] Node 0 DMA32/shadow prio:41 portion:0 scan:0 a_anon:32 643822937ms i_anon:6692 643822937ms a_file:200 129932499ms i_file:28803 130712499ms unevictable:0 reclaim_stat: 89 41 2497 88
[13732772.685664] Node 0 Normal prio:13 portion:51230 scan:0 a_anon:26317 32418ms i_anon:26354 32418ms a_file:23 32418ms i_file:0 32418ms unevictable:0 reclaim_stat: 7844 3350 140 58
[13732772.685674] Node 0 Normal/shadow prio:41 portion:0 scan:0 a_anon:4024 169652765ms i_anon:95115 169652765ms a_file:1521 77612487ms i_file:461018 77732487ms unevictable:0 reclaim_stat: 7627 6311 35568 4221
[13732772.685682] Total 658071 anon:166506 file:491565 a_anon:34355 i_anon:132151 a_file:1744 i_file:489821 unevictable:0
[13732772.685687] RAM: 65512 / 65536 [8141] SWAP: 131072 / 131072 [8110] KMEM: 18464768 / inf [0] DCSZ: 4056222 / 2147483646 [0] OOMG: 155142 / inf [3684] Dirty 260 Wback 0 Dche 2902 Prnd 10625110
[13732772.685873] Out of memory in UB: Kill process 67479 (httpd) score -864 or sacrifice child
[13732772.694544] OOM killed process 67479 (httpd) vm:232620kB, rss:97368kB, swap:9376kB
[13732772.716233] OOM killed process httpd (pid=67479, ve=50200) exited, free=40994.
[13732772.716240] UB-50200-Mem-Info:
[13732772.716248] Node 0 DMA prio:40 portion:60 scan:0 a_anon:0 10592482ms i_anon:0 10592482ms a_file:0 10592482ms i_file:0 10592482ms unevictable:0 reclaim_stat: 0 0 0 0
[13732772.716256] Node 0 DMA/shadow prio:41 portion:0 scan:0 a_anon:0 1952448ms i_anon:0 1952448ms a_file:0 1952448ms i_file:0 1952448ms unevictable:0 reclaim_stat: 0 0 0 0
[13732772.716265] Node 0 DMA32 prio:13 portion:14245 scan:0 a_anon:2463 32448ms i_anon:1869 32448ms a_file:0 32448ms i_file:1 32448ms unevictable:0 reclaim_stat: 1514 859 1 0
[13732772.716274] Node 0 DMA32/shadow prio:41 portion:0 scan:0 a_anon:32 643822967ms i_anon:5056 643822967ms a_file:200 129932529ms i_file:28803 130712529ms unevictable:0 reclaim_stat: 89 41 2497 88
[13732772.716283] Node 0 Normal prio:13 portion:51230 scan:0 a_anon:25189 32448ms i_anon:15877 32448ms a_file:3 32448ms i_file:154 32448ms unevictable:0 reclaim_stat: 8196 3631 273 32
[13732772.716292] Node 0 Normal/shadow prio:41 portion:0 scan:0 a_anon:4024 169652795ms i_anon:94969 169652795ms a_file:1525 77612517ms i_file:460879 77732517ms unevictable:0 reclaim_stat: 7627 6311 35572 4225
[13732772.716302] Total 641043 anon:149478 file:491565 a_anon:31708 i_anon:117770 a_file:1728 i_file:489837 unevictable:0
[13732772.716307] RAM: 50791 / 65536 [8141] SWAP: 129431 / 131072 [8110] KMEM: 17965056 / inf [0] DCSZ: 4056222 / 2147483646 [0] OOMG: 155142 / inf [3685] Dirty 260 Wback 0 Dche 2902 Prnd 10625110
[13732772.716346] <<< 50200 oom generation 3684 ends (task died)
[13735605.022021] >>> 50200 oom generation 3685 starts
[13735605.022031] 74815 (httpd) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
[13735605.022036] UB-50200-Mem-Info:
[13735605.022043] Node 0 DMA prio:21 portion:60 scan:0 a_anon:0 1844754ms i_anon:0 1844754ms a_file:0 1844754ms i_file:0 1844754ms unevictable:0 reclaim_stat: 0 0 0 0
[13735605.022053] Node 0 DMA/shadow prio:30 portion:0 scan:0 a_anon:0 284751ms i_anon:0 284751ms a_file:0 284751ms i_file:0 284751ms unevictable:0 reclaim_stat: 0 0 0 0
[13735605.022062] Node 0 DMA32 prio:13 portion:14245 scan:0 a_anon:586 44751ms i_anon:397 44751ms a_file:0 44751ms i_file:1 44751ms unevictable:0 reclaim_stat: 376 304 1 0
[13735605.022072] Node 0 DMA32/shadow prio:41 portion:0 scan:0 a_anon:32 646655273ms i_anon:6474 646655273ms a_file:200 132764835ms i_file:8178 133544835ms unevictable:0 reclaim_stat: 89 41 1458 2
[13735605.022083] Node 0 Normal prio:13 portion:51230 scan:145 a_anon:29393 44751ms i_anon:29438 44751ms a_file:20 44751ms i_file:6 44751ms unevictable:0 reclaim_stat: 11664 7629 1810 516
[13735605.022093] Node 0 Normal/shadow prio:41 portion:0 scan:0 a_anon:4071 172485101ms i_anon:94850 172485101ms a_file:1604 80444823ms i_file:22307 80564823ms unevictable:0 reclaim_stat: 7691 6358 5933 44
[13735605.022102] Total 197557 anon:165241 file:32316 a_anon:34082 i_anon:131159 a_file:1824 i_file:30492 unevictable:0
[13735605.022107] RAM: 65536 / 65536 [8142] SWAP: 131072 / 131072 [8111] KMEM: 18604032 / inf [0] DCSZ: 4047344 / 2147483646 [0] OOMG: 155142 / inf [3685] Dirty 130 Wback 0 Dche 2862 Prnd 10625840
[13735605.022318] Out of memory in UB: Kill process 74815 (httpd) score -876 or sacrifice child
[13735605.030859] OOM killed process 74815 (httpd) vm:224268kB, rss:94496kB, swap:3036kB
[13735605.051188] OOM killed process httpd (pid=74815, ve=50200) exited, free=36331.
[13735605.051195] UB-50200-Mem-Info:
[13735605.051202] Node 0 DMA prio:21 portion:60 scan:0 a_anon:0 1844783ms i_anon:0 1844783ms a_file:0 1844783ms i_file:0 1844783ms unevictable:0 reclaim_stat: 0 0 0 0
[13735605.051216] Node 0 DMA/shadow prio:30 portion:0 scan:0 a_anon:0 284781ms i_anon:0 284781ms a_file:0 284781ms i_file:0 284781ms unevictable:0 reclaim_stat: 0 0 0 0
[13735605.051225] Node 0 DMA32 prio:13 portion:14245 scan:0 a_anon:37 44781ms i_anon:46 44781ms a_file:0 44781ms i_file:2 44781ms unevictable:0 reclaim_stat: 381 304 2 0
[13735605.051234] Node 0 DMA32/shadow prio:41 portion:0 scan:0 a_anon:32 646655303ms i_anon:6385 646655303ms a_file:200 132764865ms i_file:8178 133544865ms unevictable:0 reclaim_stat: 89 41 1458 2
[13735605.051243] Node 0 Normal prio:13 portion:51230 scan:145 a_anon:16961 44781ms i_anon:28006 44781ms a_file:20 44781ms i_file:15 44781ms unevictable:0 reclaim_stat: 13058 9023 1819 516
[13735605.051252] Node 0 Normal/shadow prio:41 portion:0 scan:0 a_anon:4071 172485131ms i_anon:94843 172485131ms a_file:1604 80444853ms i_file:22303 80564853ms unevictable:0 reclaim_stat: 7691 6358 5933 44
[13735605.051266] Total 182703 anon:150381 file:32322 a_anon:21101 i_anon:129280 a_file:1824 i_file:30498 unevictable:0
[13735605.051271] RAM: 50798 / 65536 [8142] SWAP: 130591 / 131072 [8111] KMEM: 18063360 / inf [0] DCSZ: 4047344 / 2147483646 [0] OOMG: 155142 / inf [3686] Dirty 130 Wback 0 Dche 2862 Prnd 10625840
[13735605.051283] <<< 50200 oom generation 3685 ends (task died)
[13736335.834400] >>> 50200 oom generation 3686 starts
[13736335.834409] 77892 (httpd) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
[13736335.834413] UB-50200-Mem-Info:
[13736335.834420] Node 0 DMA prio:21 portion:60 scan:0 a_anon:0 2575567ms i_anon:0 2575567ms a_file:0 2575567ms i_file:0 2575567ms unevictable:0 reclaim_stat: 0 0 0 0
[13736335.834428] Node 0 DMA/shadow prio:30 portion:0 scan:0 a_anon:0 1015564ms i_anon:0 1015564ms a_file:0 1015564ms i_file:0 1015564ms unevictable:0 reclaim_stat: 0 0 0 0
[13736335.834437] Node 0 DMA32 prio:13 portion:14245 scan:0 a_anon:0 55564ms i_anon:10 55564ms a_file:2 55564ms i_file:0 55564ms unevictable:0 reclaim_stat: 96 30 10 2
[13736335.834446] Node 0 DMA32/shadow prio:41 portion:0 scan:0 a_anon:32 647386086ms i_anon:8110 647386086ms a_file:200 133495648ms i_file:8228 134275648ms unevictable:0 reclaim_stat: 89 41 1845 5
[13736335.834455] Node 0 Normal prio:13 portion:51230 scan:69 a_anon:30147 55564ms i_anon:30149 55564ms a_file:10 55564ms i_file:0 55564ms unevictable:0 reclaim_stat: 14938 6529 90 29
[13736335.834464] Node 0 Normal/shadow prio:41 portion:0 scan:0 a_anon:7554 173215914ms i_anon:89289 173215914ms a_file:1596 1135564ms i_file:13081 1975567ms unevictable:0 reclaim_stat: 11628 9841 2255 363
[13736335.834473] Total 188408 anon:165291 file:23117 a_anon:37733 i_anon:127558 a_file:1808 i_file:21309 unevictable:0
[13736335.834478] RAM: 65519 / 65536 [8143] SWAP: 131072 / 131072 [8112] KMEM: 19673088 / inf [0] DCSZ: 4070102 / 2147483646 [0] OOMG: 155142 / inf [3686] Dirty 130 Wback 0 Dche 2902 Prnd 10626081
[13736335.834678] Out of memory in UB: Kill process 71250 (httpd) score -886 or sacrifice child
[13736335.843169] OOM killed process 71250 (httpd) vm:218048kB, rss:86820kB, swap:3132kB
[13736335.855523] OOM killed process httpd (pid=71250, ve=50200) exited, free=1681922.
[13736335.855530] UB-50200-Mem-Info:
[13736335.855537] Node 0 DMA prio:21 portion:60 scan:0 a_anon:0 2575588ms i_anon:0 2575588ms a_file:0 2575588ms i_file:0 2575588ms unevictable:0 reclaim_stat: 0 0 0 0
[13736335.855546] Node 0 DMA/shadow prio:30 portion:0 scan:0 a_anon:0 1015585ms i_anon:0 1015585ms a_file:0 1015585ms i_file:0 1015585ms unevictable:0 reclaim_stat: 0 0 0 0
[13736335.855555] Node 0 DMA32 prio:13 portion:14245 scan:0 a_anon:0 55585ms i_anon:12 55585ms a_file:2 55585ms i_file:0 55585ms unevictable:0 reclaim_stat: 98 30 10 2
[13736335.855564] Node 0 DMA32/shadow prio:41 portion:0 scan:0 a_anon:32 647386107ms i_anon:8107 647386107ms a_file:200 133495669ms i_file:8228 134275669ms unevictable:0 reclaim_stat: 89 41 1845 5
[13736335.855573] Node 0 Normal prio:13 portion:51230 scan:160 a_anon:25483 55585ms i_anon:21785 55585ms a_file:16 55585ms i_file:0 55585ms unevictable:0 reclaim_stat: 18329 9859 82 30
[13736335.855582] Node 0 Normal/shadow prio:41 portion:0 scan:0 a_anon:7553 173215935ms i_anon:89094 173215935ms a_file:1620 1135585ms i_file:13058 1975588ms unevictable:0 reclaim_stat: 11628 9841 2279 387
[13736335.855592] Total 175190 anon:152066 file:23124 a_anon:33068 i_anon:118998 a_file:1838 i_file:21286 unevictable:0
[13736335.855597] RAM: 52784 / 65536 [8143] SWAP: 130573 / 131072 [8112] KMEM: 19156992 / inf [0] DCSZ: 4070102 / 2147483646 [0] OOMG: 155142 / inf [3687] Dirty 130 Wback 0 Dche 2902 Prnd 10626081
[13736335.855609] <<< 50200 oom generation 3686 ends (task died)
[13736348.696196] >>> 50200 oom generation 3687 starts
[13736348.696206] 77883 (httpd) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
[13736348.696210] UB-50200-Mem-Info:
[13736348.696217] Node 0 DMA prio:21 portion:60 scan:0 a_anon:0 2588429ms i_anon:0 2588429ms a_file:0 2588429ms i_file:0 2588429ms unevictable:0 reclaim_stat: 0 0 0 0
[13736348.696227] Node 0 DMA/shadow prio:31 portion:0 scan:0 a_anon:0 1028426ms i_anon:0 1028426ms a_file:0 1028426ms i_file:0 1028426ms unevictable:0 reclaim_stat: 0 0 0 0
[13736348.696236] Node 0 DMA32 prio:13 portion:14245 scan:0 a_anon:0 8426ms i_anon:5 8426ms a_file:3 8426ms i_file:1 8426ms unevictable:0 reclaim_stat: 88 22 21 5
[13736348.696246] Node 0 DMA32/shadow prio:41 portion:0 scan:0 a_anon:32 647398948ms i_anon:8119 647398948ms a_file:200 133508510ms i_file:8227 134288510ms unevictable:0 reclaim_stat: 89 41 1845 5
[13736348.696269] Node 0 Normal prio:13 portion:51230 scan:114 a_anon:30284 8426ms i_anon:30241 8426ms a_file:0 8426ms i_file:0 8426ms unevictable:0 reclaim_stat: 8599 2503 23 7
[13736348.696283] Node 0 Normal/shadow prio:41 portion:0 scan:0 a_anon:7553 173228776ms i_anon:89230 173228776ms a_file:1620 1148426ms i_file:13186 1988429ms unevictable:0 reclaim_stat: 11628 9841 2279 387
[13736348.696298] Total 188701 anon:165464 file:23237 a_anon:37869 i_anon:127595 a_file:1823 i_file:21414 unevictable:0
[13736348.696308] RAM: 65530 / 65536 [8144] SWAP: 131072 / 131072 [8113] KMEM: 19677184 / inf [0] DCSZ: 4061718 / 2147483646 [0] OOMG: 155142 / inf [3687] Dirty 130 Wback 0 Dche 2899 Prnd 10626081
[13736348.696532] Out of memory in UB: Kill process 75363 (httpd) score -881 or sacrifice child
[13736348.705001] OOM killed process 75363 (httpd) vm:232128kB, rss:81476kB, swap:11768kB
[13736348.718455] OOM killed process httpd (pid=75363, ve=50200) exited, free=1684832.
[13736348.718462] UB-50200-Mem-Info:
[13736348.718469] Node 0 DMA prio:21 portion:60 scan:0 a_anon:0 2588451ms i_anon:0 2588451ms a_file:0 2588451ms i_file:0 2588451ms unevictable:0 reclaim_stat: 0 0 0 0
[13736348.718477] Node 0 DMA/shadow prio:31 portion:0 scan:0 a_anon:0 1028448ms i_anon:0 1028448ms a_file:0 1028448ms i_file:0 1028448ms unevictable:0 reclaim_stat: 0 0 0 0
[13736348.718486] Node 0 DMA32 prio:13 portion:14245 scan:0 a_anon:0 8448ms i_anon:5 8448ms a_file:3 8448ms i_file:1 8448ms unevictable:0 reclaim_stat: 88 22 21 5
[13736348.718494] Node 0 DMA32/shadow prio:41 portion:0 scan:0 a_anon:32 647398970ms i_anon:5996 647398970ms a_file:200 133508532ms i_file:8227 134288532ms unevictable:0 reclaim_stat: 89 41 1845 5
[13736348.718504] Node 0 Normal prio:13 portion:51230 scan:114 a_anon:19828 8448ms i_anon:26106 8448ms a_file:1 8448ms i_file:7 8448ms unevictable:0 reclaim_stat: 8792 2668 31 8
[13736348.718513] Node 0 Normal/shadow prio:41 portion:0 scan:0 a_anon:7553 173228798ms i_anon:89042 173228798ms a_file:1711 1148448ms i_file:13082 1988451ms unevictable:0 reclaim_stat: 11628 9841 2370 478
[13736348.718522] Total 171794 anon:148562 file:23232 a_anon:27413 i_anon:121149 a_file:1915 i_file:21317 unevictable:0
[13736348.718527] RAM: 51754 / 65536 [8144] SWAP: 128812 / 131072 [8113] KMEM: 19161088 / inf [0] DCSZ: 4061718 / 2147483646 [0] OOMG: 155142 / inf [3688] Dirty 130 Wback 0 Dche 2899 Prnd 10626081
[13736348.718539] <<< 50200 oom generation 3687 ends (task died)
[13737437.173423] >>> 50200 oom generation 3688 starts
[13737437.173432] 81018 (httpd) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
[13737437.173449] UB-50200-Mem-Info:
[13737437.173459] Node 0 DMA prio:22 portion:60 scan:0 a_anon:0 3676906ms i_anon:0 3676906ms a_file:0 3676906ms i_file:0 3676906ms unevictable:0 reclaim_stat: 0 0 0 0
[13737437.173476] Node 0 DMA/shadow prio:33 portion:0 scan:0 a_anon:0 2116903ms i_anon:0 2116903ms a_file:0 2116903ms i_file:0 2116903ms unevictable:0 reclaim_stat: 0 0 0 0
[13737437.173493] Node 0 DMA32 prio:13 portion:14245 scan:0 a_anon:0 16903ms i_anon:13 16903ms a_file:11 16903ms i_file:1 16903ms unevictable:0 reclaim_stat: 81 26 58 20
[13737437.173510] Node 0 DMA32/shadow prio:41 portion:0 scan:0 a_anon:32 648487425ms i_anon:2372 648487425ms a_file:200 134596987ms i_file:8202 135376987ms unevictable:0 reclaim_stat: 89 41 1845 5
[13737437.173525] Node 0 Normal prio:13 portion:51230 scan:0 a_anon:30463 16903ms i_anon:30491 16903ms a_file:0 16903ms i_file:28 16903ms unevictable:0 reclaim_stat: 7932 3743 1334 365
[13737437.173541] Node 0 Normal/shadow prio:41 portion:0 scan:0 a_anon:7488 174317253ms i_anon:95377 174317253ms a_file:1340 2236903ms i_file:28025 3076906ms unevictable:0 reclaim_stat: 11692 9904 2400 508
[13737437.173558] Total 204043 anon:166236 file:37807 a_anon:37983 i_anon:128253 a_file:1551 i_file:36256 unevictable:0
[13737437.173568] RAM: 65536 / 65536 [8145] SWAP: 131072 / 131072 [8114] KMEM: 18427904 / inf [0] DCSZ: 3979138 / 2147483646 [0] OOMG: 155142 / inf [3688] Dirty 195 Wback 0 Dche 2887 Prnd 10627349
[13737437.173801] Out of memory in UB: Kill process 81012 (httpd) score -871 or sacrifice child
[13737437.182253] OOM killed process 81012 (httpd) vm:241832kB, rss:35848kB, swap:65420kB
[13737437.196645] OOM killed process httpd (pid=81012, ve=50200) exited, free=1069563.
[13737437.196652] UB-50200-Mem-Info:
[13737437.196660] Node 0 DMA prio:22 portion:60 scan:0 a_anon:0 3676929ms i_anon:0 3676929ms a_file:0 3676929ms i_file:0 3676929ms unevictable:0 reclaim_stat: 0 0 0 0
[13737437.196668] Node 0 DMA/shadow prio:33 portion:0 scan:0 a_anon:0 2116926ms i_anon:0 2116926ms a_file:0 2116926ms i_file:0 2116926ms unevictable:0 reclaim_stat: 0 0 0 0
[13737437.196678] Node 0 DMA32 prio:13 portion:14245 scan:0 a_anon:0 16926ms i_anon:16 16926ms a_file:11 16926ms i_file:5 16926ms unevictable:0 reclaim_stat: 84 26 62 20
[13737437.196686] Node 0 DMA32/shadow prio:41 portion:0 scan:0 a_anon:32 648487448ms i_anon:2368 648487448ms a_file:200 134597010ms i_file:8194 135377010ms unevictable:0 reclaim_stat: 89 41 1845 5
[13737437.196696] Node 0 Normal prio:13 portion:51230 scan:0 a_anon:30419 16926ms i_anon:27645 16926ms a_file:1 16926ms i_file:95 16926ms unevictable:0 reclaim_stat: 8101 3883 1402 366
[13737437.196705] Node 0 Normal/shadow prio:41 portion:0 scan:0 a_anon:7488 174317276ms i_anon:79629 174317276ms a_file:1340 2236926ms i_file:27952 3076929ms unevictable:0 reclaim_stat: 11692 9904 2400 508
[13737437.196713] Total 185395 anon:147597 file:37798 a_anon:37939 i_anon:109658 a_file:1552 i_file:36246 unevictable:0
[13737437.196719] RAM: 63028 / 65536 [8145] SWAP: 115535 / 131072 [8114] KMEM: 17924096 / inf [0] DCSZ: 3979138 / 2147483646 [0] OOMG: 155142 / inf [3689] Dirty 195 Wback 0 Dche 2887 Prnd 10627349
[13737437.196731] <<< 50200 oom generation 3688 ends (task died)
[13738489.851468] >>> 50200 oom generation 3689 starts
[13738489.851478] 81018 (httpd) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
[13738489.851482] UB-50200-Mem-Info:
[13738489.851489] Node 0 DMA prio:25 portion:60 scan:0 a_anon:0 4729584ms i_anon:0 4729584ms a_file:0 4729584ms i_file:0 4729584ms unevictable:0 reclaim_stat: 0 0 0 0
[13738489.851498] Node 0 DMA/shadow prio:36 portion:0 scan:0 a_anon:0 3169581ms i_anon:0 3169581ms a_file:0 3169581ms i_file:0 3169581ms unevictable:0 reclaim_stat: 0 0 0 0
[13738489.851506] Node 0 DMA32 prio:13 portion:14245 scan:0 a_anon:277 49578ms i_anon:254 49578ms a_file:0 49578ms i_file:2 49578ms unevictable:0 reclaim_stat: 314 254 2 0
[13738489.851515] Node 0 DMA32/shadow prio:41 portion:0 scan:0 a_anon:32 649540103ms i_anon:7800 649540103ms a_file:201 135649665ms i_file:8168 136429665ms unevictable:0 reclaim_stat: 89 41 1245 1
[13738489.851524] Node 0 Normal prio:13 portion:51230 scan:186 a_anon:30070 49578ms i_anon:30058 49578ms a_file:17 49578ms i_file:14 49578ms unevictable:0 reclaim_stat: 11606 7275 15213 4342
[13738489.851534] Node 0 Normal/shadow prio:32 portion:0 scan:0 a_anon:1138 1249581ms i_anon:95572 1729581ms a_file:1364 709578ms i_file:13403 769579ms unevictable:0 reclaim_stat: 20788 14720 2262 661
[13738489.851542] Total 188370 anon:165201 file:23169 a_anon:31517 i_anon:133684 a_file:1582 i_file:21587 unevictable:0
[13738489.851547] RAM: 65528 / 65536 [8146] SWAP: 131072 / 131072 [8115] KMEM: 18612224 / inf [0] DCSZ: 3988598 / 2147483646 [0] OOMG: 155142 / inf [3689] Dirty 65 Wback 0 Dche 2854 Prnd 10628043
[13738489.851737] Out of memory in UB: Kill process 81018 (httpd) score -870 or sacrifice child
[13738489.860203] OOM killed process 81018 (httpd) vm:229916kB, rss:98172kB, swap:4120kB
[13738489.879514] OOM killed process httpd (pid=81018, ve=50200) exited, free=39369.
[13738489.879520] UB-50200-Mem-Info:
[13738489.879527] Node 0 DMA prio:25 portion:60 scan:0 a_anon:0 4729612ms i_anon:0 4729612ms a_file:0 4729612ms i_file:0 4729612ms unevictable:0 reclaim_stat: 0 0 0 0
[13738489.879536] Node 0 DMA/shadow prio:36 portion:0 scan:0 a_anon:0 3169609ms i_anon:0 3169609ms a_file:0 3169609ms i_file:0 3169609ms unevictable:0 reclaim_stat: 0 0 0 0
[13738489.879544] Node 0 DMA32 prio:13 portion:14245 scan:0 a_anon:12 49606ms i_anon:12 49606ms a_file:0 49606ms i_file:2 49606ms unevictable:0 reclaim_stat: 314 254 2 0
[13738489.879553] Node 0 DMA32/shadow prio:41 portion:0 scan:0 a_anon:32 649540131ms i_anon:7525 649540131ms a_file:201 135649693ms i_file:8168 136429693ms unevictable:0 reclaim_stat: 89 41 1245 1
[13738489.879562] Node 0 Normal prio:13 portion:51230 scan:186 a_anon:20072 49606ms i_anon:24642 49606ms a_file:17 49606ms i_file:34 49606ms unevictable:0 reclaim_stat: 11726 7395 15233 4342
[13738489.879571] Node 0 Normal/shadow prio:32 portion:0 scan:0 a_anon:1138 1249609ms i_anon:95411 1729609ms a_file:1364 709606ms i_file:13402 769607ms unevictable:0 reclaim_stat: 20788 14720 2262 661
[13738489.879581] Total 172032 anon:148844 file:23188 a_anon:21254 i_anon:127590 a_file:1582 i_file:21606 unevictable:0
[13738489.879586] RAM: 49775 / 65536 [8146] SWAP: 130560 / 131072 [8115] KMEM: 18092032 / inf [0] DCSZ: 3988598 / 2147483646 [0] OOMG: 155142 / inf [3690] Dirty 65 Wback 0 Dche 2854 Prnd 10628043
[13738489.879598] <<< 50200 oom generation 3689 ends (task died)
[13747953.818727] >>> 50200 oom generation 3690 starts
[13747953.818737] 137946 (httpd) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
[13747953.818741] UB-50200-Mem-Info:
[13747953.818748] Node 0 DMA prio:41 portion:60 scan:0 a_anon:0 14193551ms i_anon:0 14193551ms a_file:0 14193551ms i_file:0 14193551ms unevictable:0 reclaim_stat: 0 0 0 0
[13747953.818757] Node 0 DMA/shadow prio:41 portion:0 scan:0 a_anon:0 7473545ms i_anon:0 7473545ms a_file:0 7473545ms i_file:0 7473545ms unevictable:0 reclaim_stat: 0 0 0 0
[13747953.818765] Node 0 DMA32 prio:13 portion:14245 scan:0 a_anon:18 33540ms i_anon:29 33540ms a_file:0 33540ms i_file:1 33540ms unevictable:0 reclaim_stat: 98 60 1 0
[13747953.818774] Node 0 DMA32/shadow prio:41 portion:0 scan:0 a_anon:32 659004070ms i_anon:2301 659004070ms a_file:201 145113632ms i_file:7996 145893632ms unevictable:0 reclaim_stat: 89 41 1183 8
[13747953.818783] Node 0 Normal prio:14 portion:51230 scan:121 a_anon:30105 33540ms i_anon:30126 33540ms a_file:25 33540ms i_file:0 33540ms unevictable:0 reclaim_stat: 11321 6069 143 68
[13747953.818793] Node 0 Normal/shadow prio:41 portion:0 scan:0 a_anon:826 10713548ms i_anon:101923 11193548ms a_file:1672 10173545ms i_file:40553 10233546ms unevictable:0 reclaim_stat: 20788 14720 2780 1179
[13747953.818802] Total 215808 anon:165360 file:50448 a_anon:30981 i_anon:134379 a_file:1898 i_file:48550 unevictable:0
[13747953.818806] RAM: 65536 / 65536 [8147] SWAP: 131072 / 131072 [8116] KMEM: 18743296 / inf [0] DCSZ: 4080319 / 2147483646 [0] OOMG: 155142 / inf [3690] Dirty 130 Wback 0 Dche 2897 Prnd 10634032
[13747953.818993] Out of memory in UB: Kill process 137946 (httpd) score -872 or sacrifice child
[13747953.827283] OOM killed process 137946 (httpd) vm:226200kB, rss:97844kB, swap:3012kB
[13747953.847506] OOM killed process httpd (pid=137946, ve=50200) exited, free=988432.
[13747953.847512] UB-50200-Mem-Info:
[13747953.847519] Node 0 DMA prio:41 portion:60 scan:0 a_anon:0 14193580ms i_anon:0 14193580ms a_file:0 14193580ms i_file:0 14193580ms unevictable:0 reclaim_stat: 0 0 0 0
[13747953.847528] Node 0 DMA/shadow prio:41 portion:0 scan:0 a_anon:0 7473574ms i_anon:0 7473574ms a_file:0 7473574ms i_file:0 7473574ms unevictable:0 reclaim_stat: 0 0 0 0
[13747953.847537] Node 0 DMA32 prio:13 portion:14245 scan:0 a_anon:18 33569ms i_anon:30 33569ms a_file:0 33569ms i_file:1 33569ms unevictable:0 reclaim_stat: 99 60 1 0
[13747953.847545] Node 0 DMA32/shadow prio:41 portion:0 scan:0 a_anon:32 659004099ms i_anon:2300 659004099ms a_file:201 145113661ms i_file:7996 145893661ms unevictable:0 reclaim_stat: 89 41 1183 8
[13747953.847555] Node 0 Normal prio:14 portion:51230 scan:121 a_anon:29687 33569ms i_anon:15395 33569ms a_file:25 33569ms i_file:11 33569ms unevictable:0 reclaim_stat: 15905 10626 154 68
[13747953.847564] Node 0 Normal/shadow prio:41 portion:0 scan:0 a_anon:826 10713577ms i_anon:101717 11193577ms a_file:1672 10173574ms i_file:40549 10233575ms unevictable:0 reclaim_stat: 20788 14720 2780 1179
[13747953.847573] Total 200460 anon:150005 file:50455 a_anon:30563 i_anon:119442 a_file:1898 i_file:48557 unevictable:0
[13747953.847578] RAM: 50553 / 65536 [8147] SWAP: 130738 / 131072 [8116] KMEM: 18251776 / inf [0] DCSZ: 4080319 / 2147483646 [0] OOMG: 155142 / inf [3691] Dirty 130 Wback 0 Dche 2897 Prnd 10634032
[13747953.847590] <<< 50200 oom generation 3690 ends (task died)


vzcfgvalidate /etc/vz/conf/50200.conf
Error: physpages.bar should be = 0 (currently, 1000000)
[root@box13 vz]# vzctl set 50200 --physpages.bar 0 --save
vzctl: unrecognized option '--physpages.bar'

vi /etc/vz/conf/50200.conf
PHYSPAGES="0"
didn't work well

solution is to add some missing options (the issue is mainly that scripts made for older Centos/OpenVZ kernels such as Centos 5 will cause issues seen above until parameters below are added to the config)
try
PHYSPAGES="0:unlimited"
KMEMSIZE="1147483646:1947483646"
NUMIPTENT="999999"
OOMGUARPAGES="1000000"


 


Tags:

openvz, oom, centos, config, settings, solutionlinux, smp, mar, msk, _, gnu, linux, privvmpages, quot, containers, kernel, users, buffers, cached, mem, cache, swap, anon, a_anon, i_anon, a_file, i_file, unevictable, ram, kmem, inf, dcsz, oomg, wback, dche, prnd, ub, httpd, vm, kb, rss, pid, ve, exited, info, node, dma, prio, portion, scan, reclaim_stat, task, invoked, loc, gfp, oomkilladj, vzcfgvalidate, etc, vz, conf, physpages, currently, vzctl, unrecognized, vi, didn, mainly, scripts, kernels, parameters, unlimited, kmemsize, numiptent, oomguarpages,

Latest Articles

  • How high can a Xeon CPU get?
  • bash fix PATH environment variable "command not found" solution
  • Ubuntu Linux Mint Debian Redhat Youtube Cannot Play HD or 4K videos, dropped frames or high CPU usage with Nvidia or AMD Driver
  • hostapd example configuration for high speed AC on 5GHz using WPA2
  • hostapd how to enable and use WPS to connect wireless devices like printers
  • Dell Server Workstation iDRAC Dead after Firmware Update Solution R720, R320, R730
  • Cloned VM/Server/Computer in Linux won't boot and goes to initramfs busybox Solution
  • How To Add Windows 7 8 10 11 to GRUB Boot List Dual Booting
  • How to configure OpenDKIM on Linux with Postfix and setup bind zonefile
  • Debian Ubuntu 10/11/12 Linux how to get tftpd-hpa server setup tutorial
  • efibootmgr: option requires an argument -- 'd' efibootmgr version 15 grub-install.real: error: efibootmgr failed to register the boot entry: Operation not permitted.
  • Apache Error Won't start SSL Cert Issue Solution Unable to configure verify locations for client authentication SSL Library Error: 151441510 error:0906D066:PEM routines:PEM_read_bio:bad end line SSL Library Error: 185090057 error:0B084009:x509 certif
  • Linux Debian Mint Ubuntu Bridge br0 gets random IP
  • redis requirements
  • How to kill a docker swarm
  • docker swarm silly issues
  • isc-dhcp-server dhcpd how to get longer lease
  • nvidia cannot resume from sleep Comm: nvidia-sleep.sh Tainted: Linux Ubuntu Mint Debian
  • zfs and LUKS how to recover in Linux
  • [error] (28)No space left on device: Cannot create SSLMutex Apache Solution Linux CentOS Ubuntu Debian Mint