Linux Kernel OOM Log Analysis

Sample oom log:

[2-3219.1870] vd-net-config invoked oom-killer: gfp_mask=0x80d0, order=0, oom_score_adj=-900

[2-3219.1870] CPU: 2 PID: 467 Comm: vd-net-config Tainted: PO 4.1.10 #1

[2-3219.1870] SCHED_NORMAL (p:120, static_p:120, normal_p:120, rt_p:0)

[2-3219.1870] Hardware name: Samsung SDP1601(Flattened Device Tree)

[2-3219.1870] Backtrace:

[2-3219.1870] [<c0014200>] (dump_backtrace) from [<c0015020>] (show_stack+0x20/0x28)

[2-3219.1870]  r7:600e0013 r6:c08f8ab8 r5:c08cddf0 r4:00000000

[2-3219.1870] [<c0015000>] (show_stack) from [<c062fcd8>] (dump_stack+0x140/0x198)

[2-3219.1870] [<c062fb98>] (dump_stack) from [<c013885c>] (dump_header.isra.17+0xb0/0x144)

[2-3219.1870]  r10:c08d217c r9:00000000 r8:00000000 r7:000080d0 r6:00000000 r5:c4932bc0

[2-3219.1870]  r4:c08e1b1c

[2-3219.1871] [<c01387ac>] (dump_header.isra.17) from [<c0139324>] (oom_kill_process+0x1f8/0x4e4)

[2-3219.1871]  r8:00000000 r7:000080d0 r6:000003f4 r5:c4932ff0 r4:c4932bc0

[2-3219.1871] [<c013912c>] (oom_kill_process) from [<c0139a2c>] (__out_of_memory.isra.21+0x41c/0x558)

[2-3219.1871]  r10:c08d217c r9:00000000 r8:00000000 r7:000080d0 r6:000c14e7 r5:00000000

[2-3219.1871]  r4:c08e1b1c

[2-3219.1871] [<c0139610>] (__out_of_memory.isra.21) from [<c0139d64>] (out_of_memory+0x5c/0x74)

[2-3219.1871]  r10:00000040 r9:d412a000 r8:00000000 r7:00000000 r6:00000000 r5:00000000

[2-3219.1871]  r4:00000001

[2-3219.1871] [<c0139d08>] (out_of_memory) from [<c013f734>] (__alloc_pages_nodemask+0x960/0xa2c)

[2-3219.1871]  r7:000080d0 r6:00000001 r5:000080d0 r4:00000000

[2-3219.1871] [<c013edd4>] (__alloc_pages_nodemask) from [<c013f820>] (__get_free_pages+0x20/0x3c)

[2-3219.1871]  r10:00002000 r9:cb66c668 r8:00000000 r7:d412beb0 r6:00000000 r5:d412a000

[2-3219.1871]  r4:cb66c668

[2-3219.1871] [<c013f800>] (__get_free_pages) from [<c013f85c>] (get_zeroed_page+0x20/0x28)

[2-3219.1871] [<c013f83c>] (get_zeroed_page) from [<c01fdb48>] (kernfs_iop_follow_link+0x28/0x1ac)

[2-3219.1871] [<c01fdb20>] (kernfs_iop_follow_link) from [<c019a460>] (link_path_walk+0x450/0x8d4)

[2-3219.1872]  r10:00002000 r9:cb66c668 r8:00000000 r7:c38c53e5 r6:00000000 r5:d412a000

[2-3219.1872]  r4:d412beb0

[2-3219.1872] [<c019a010>] (link_path_walk) from [<c019aa60>] (path_init+0x17c/0x3f8)

[2-3219.1872]  r10:00002000 r9:00000000 r8:00000010 r7:d412beb0 r6:c38c53d0 r5:d412bec8

[2-3219.1872]  r4:d412beb0

[2-3219.1872] [<c019a8e4>] (path_init) from [<c019ad00>] (path_lookupat+0x24/0x750)

[2-3219.1872]  r9:00000000 r8:00000010 r7:d412beb0 r6:c38c53c0 r5:00000041 r4:d412beb0

[2-3219.1872] [<c019acdc>] (path_lookupat) from [<c019b458>] (filename_lookup+0x2c/0x7c)

[2-3219.1872]  r10:00002000 r9:00000000 r8:00000010 r7:d412beb0 r6:c38c53c0 r5:ffffff9c

[2-3219.1872]  r4:00000001

[2-3219.1872] [<c019b42c>] (filename_lookup) from [<c019ca88>] (user_path_at_empty+0x64/0x90)

[2-3219.1872]  r7:d412bf60 r6:ffffff9c r5:00000001 r4:c38c53c0

[2-3219.1872] [<c019ca24>] (user_path_at_empty) from [<c019cae0>] (user_path_at+0x2c/0x50)

[2-3219.1872]  r7:00057fbc r6:ffffff9c r5:d412bf60 r4:00057fbc

[2-3219.1872] [<c019cab4>] (user_path_at) from [<c018bbd8>] (SyS_faccessat+0xa0/0x200)

[2-3219.1872]  r5:c91acdc0 r4:00000001

[2-3219.1873] [<c018bb38>] (SyS_faccessat) from [<c018bd5c>] (SyS_access+0x24/0x2c)

[2-3219.1873]  r10:00002000 r9:d412a000 r8:c0010024 r7:00000021 r6:00000000 r5:00073940

[2-3219.1873]  r4:00071840

[2-3219.1873] [<c018bd38>] (SyS_access) from [<c0

[read  2017-03-20 04:46:07.208233] 010000>] (__sys_trace_return+0x0/0x20)

[2-3219.1873] Mem-Info:

[2-3219.1875] active_anon:148141 inactive_anon:52931 isolated_anon:0

[2-3219.1875]  active_file:29121 inactive_file:30395 isolated_file:50

[2-3219.1875]  unevictable:2 dirty:17 writeback:0 unstable:0

[2-3219.1875]  slab_reclaimable:15062 slab_unreclaimable:54733

[2-3219.1875]  mapped:24034 shmem:7265 pagetables:10040 bounce:0

[2-3219.1875]  free:20647 free_pcp:0 free_cma:16100

[2-3219.1875] Normal free:67368kB min:3032kB low:3788kB high:4548kB active_anon:0kB inactive_anon:24kB active_file:148kB inactive_file:160kB unevictable:0kB isolated(anon):0kB isolated(file):200kB present:622592kB managed:576420kB mlocked:0kB dirty:12kB writeback:4kB mapped:316kB shmem:0kB slab_reclaimable:60248kB slab_unreclaimable:218932kB kernel_stack:14176kB pagetables:40160kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:64400kB writeback_tmp:0kB pages_scanned:6352 all_unreclaimable? yes

[2-3219.1875] lowmem_reserve[]: 0 94 94

[2-3219.1875] HighMem free:15220kB min:512kB low:2540kB high:4568kB active_anon:592564kB inactive_anon:211700kB active_file:116336kB inactive_file:121420kB unevictable:8kB isolated(anon):0kB isolated(file):0kB present:1542144kB managed:1542140kB mlocked:8kB dirty:56kB writeback:0kB mapped:95820kB shmem:29060kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no

[2-3219.1875] lowmem_reserve[]: 0 0 0

[2-3219.1875] VmallocUsed:      221008 kB [    8192 kB vm_map_ram,   187984 kB ioremap,    21972 kB vmalloc,      212 kB vmap,     2656 kB usermap,     8196 kB vpages,     8204 kB overlapped]

[2-3219.1875] Normal: 75*4kB (UERC) 53*8kB (URC) 84*16kB (UEMC) 32*32kB (RC) 24*64kB (C) 15*128kB (RC) 8*256kB (RC) 1*512kB (R) 3*1024kB (RC) 1*2048kB (C) 13*4096kB (C) = 67476kB

[2-3219.1876] HighMem: 2547*4kB (UMR) 208*8kB (UMR) 28*16kB (UM) 11*32kB (UMR) 25*64kB (M) 8*128kB (UMR) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15276kB

[2-3219.1876] 71761 total pagecache pages

[2-3219.1876] 4915 pages in swap cache

[2-3219.1876] Swap cache stats: add 163067, delete 158152, find 9884/23756

[2-3219.1876] Free swap  = 607248kB

[2-3219.1876] Total swap = 1048572kB

[2-3219.1876] 541184 pages RAM

[2-3219.1876] 385536 pages HighMem/MovableOnly

[2-3219.1876] 4294962456 pages reserved

[2-3219.1876] 16384 pages cma reserved

[2-3219.1892] * Sum of rss    : 1239568 (kB)

[2-3219.1892] * Sum of maxrss : 3555856 (kB)

[2-3219.1892] * Sum of swap   : 421956 (kB)

[2-3219.1892] slabinfo – version: 2.1

[2-3219.1892] # name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>

[2-3219.1892] pid_2                  5     16    256   16    1 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1892] ecryptfs_key_tfm_cache      0      0    320   25    2 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1893] ecryptfs_global_auth_tok_cache      2     17    960   17    4 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1893] ecryptfs_key_sig_cache      0      0    224   18    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1893] ecryptfs_key_record_cache      0      0    704   23    4 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1893] ecryptfs_xattr_cache      0      0   4288    7    8 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1893] ecryptfs_headers       0      0   4288    7    8 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1893] ecryptfs_sb_cache      1     18    896   18    4 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1893] ecryptfs_inode_cache      1     16   1024   16    4 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1893] ecryptfs_dentry_info_cache      1     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1893] ecryptfs_file_cache      0      0    184   22    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1893] ecryptfs_auth_tok_list_item      0      0   1024   16    4 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1894] scsi_sense_cache       1     25    320   25    2 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1894] scsi_cmd_cache         1     18    448   18    2 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1894] dp_sw_mix_q2           0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1894] dp_sw_mix_q1           0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1894] aeCbQ                  0     36    216   18    1 : tunables    0    0    0 : slabdata      2      2      0

[2-3219.1894] aePcmDumpQ             0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1895] aeMsgQMP0              0     19    208   19    1 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1895] aioCbQ                 0      0    224   18    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1895] vbi_isr_q              0     32    256   16    1 : tunables    0    0    0 : slabdata      2      2      0

[2-3219.1895] NSRPCTransaction_s      0     21    376   21    2 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1895] tzmem_registration      8     18    216   18    1 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1895] tzio_context           0     16    248   16    1ail(249) > Could not get avail at PCM handle 0xb0e00b28 : -32

[ : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1895] tzdev_page             1     22    184   22    1 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1895] kdbus_debug_stat       0      0    224   18    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1895] ip6-frags              0      0    328   24    2 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1895] fib6_nodes             5     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1896] ip6_dst_cache          4     36    448   18    2 : tunables    0    0    0 : slabdata      2      2      0

[2-3219.1896] ip6_mrt_cache          0      0    320   25    2 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1896] PINGv6                 0      0   1088   30    8 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1896] RAWv6                  8     30   1088   30    8 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1896] UDPLITEv6              0      0   1088   30    8 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1896] UDPv6                 12     30   1088   30    8 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1896] tw_sock_TCPv6          0      0    352   23    2 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1896] request_sock_TCPv6      0     20    392   20    2 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1896] TCPv6                  9     16   1984   16    8 : tunables    0    0    0 : slabdata      1      1      0

[read  2017-03-20 04:46:09.211637]

[2-3219.1897] nf_conntrack_c0918940   5020   6500    408   20    2save_error_log]  : tunables    0    0    0 : slabdata    325    325      0

[2-3219.1897] nf_conntrack_expect      0      0    336   24    2 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1897] sd_ext_cdb             2     19    208   19    1 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1897] zs_handle         109680 109687    176   23    1 : tunables    0    0    0 : slabdata   4769   4769      0

[2-3219.1897] mqueue_inode_cache      6     19    832   19    4 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1897] vdfs_fsm_node          0      0    208   19    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1898] emmcfs_exttree_key      0      0    224   18    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1898] vdfs_btree_todo_list      0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1898] vdfs_btree_record_info      0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1898] vdfs_btree_traverse_stack      0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1898] vdfs_icache            0      0   1088   30    8 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1898] vdfs4_fsm_node        46     76    208   19    1 : tunables    0    0    0 : slabdata      4      4      0

[2-3219.1898] vdfs4_exttree_key      0     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1898] vdfs4_btree_todo_list      0     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1898] vdfs4_btree_record_info      0     84    192   21    1 : tunables    0    0    0 : slabdata      4      4      0

[2-3219.1898] vdfs4_btree_traverse_stack      0     42    192   21    1 : tunables    0    0    0 : slabdata      2      2      0

[2-3219.1899] vdfs4_icache        2349   5916    960   17    4 : tunables    0    0    0 : slabdata    348    348      0

[2-3219.1899] fuse_request           0     17    480   17    2 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1899] fuse_inode             1     21    768   21    4 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1900] fat_inode_cache        5     88    720   22    4 : tunables    0    0    0 : slabdata      4      4      0

[2-3219.1900] fat_cache              1     84    192   21    1 : tunables    0    0    0 : slabdata      4      4      0

[2-3219.1900] squashfs_inode_cache    101    322    704   23    4 : tunables    0    0    0 : slabdata     14     14      0

[2-3219.1900] jbd2_transaction_s      0      0    384   21    2 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1900] jbd2_inode             0      0    200   20    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1900] jbd2_journal_handle      0      0    208   19    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1900] jbd2_journal_head      0      0    232   17    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1900] jbd2_revoke_table_s      0      0    184   22    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1900] jbd2_revoke_record_s      0      0    224   18    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1900] ext4_inode_cache       0      0   1040   31    8 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1901] ext4_free_data         0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1901] ext4_allocation_context      0      0    280   29    2 : tunabl

[read  2017-03-20 04:46:10.212981] es    0    0    0 : slabdata      0      0      0

[2-3219.1901] ext4_prealloc_space      0      0    264   31    2 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1901] ext4_system_zone       0      0    208   19    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1901] ext4_io_end            0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1901] ext4_extent_status      0      0    208   19    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1901] kioctx                 0      0    704   23    4 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1901] aio_kiocb              0      0    256   16    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1901] inotify_inode_mark    257    288    248   16    1 : tunables    0    0    0 : slabdata     18     18      0

[2-3219.1901] dio                    0      0    576   28    4 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1901] fasync_cache           0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1902] kdml_module_type      77     87    280   29    2 : tunables    0    0    0 : slabdata      3      3      0

[2-3219.1902] kdml_summary_type      0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1902] kdml_rb_type           0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1902] pid_namespace          1     29    280   29    2 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1902] posix_timers_cache      1     23    344   23    2 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1902] UNIX                1126   1156    960   17    4 : tunables    0    0    0 : slabdata     68     68      0

[2-3219.1902] ip4-frags              0      0    304   26    2 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1902] UDP-Lite               0      0    960   17    4 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1902] tcp_bind_bucket      851   1044    224   18    1 : tunables    0    0    0 : slabdata     58     58      0

[2-3219.1903] inet_peer_cache        5     25    320   25    2 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1903] secpath_cache          0      0    224   18    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1903] flow_cache             0      0    256   16    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1903] xfrm_dst_cache         0      0    512   16    2 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1903] ip_fib_trie           10     19    208   19    1 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1903] ip_fib_alias          12     20    200   20    1 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1903] ip_dst_cache          23    100    320   25    2 : tunables    0    0    0 : slabdata      4      4      0

[2-3219.1903] PING                   0      0    960   17    4 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1903] RAW                    9     17    960   17    4 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1903] UDP                   35     68    960   17    4 : tunables    0    0    0 : slabdata      4      4      0

[2-3219.1904] tw_sock_TCP         1295   1357    352   23    2 : tunables    0    0    0 : slabdata     59     59      0

[2-3219.1904] request_sock_TCP       2     80    392   20    2 : tunables    0    0    0 : slabdata      4      4      0

[2-3219.1904] TCP                  138    170   1856   17    8 : tunables    0    0    0 : slabdata     10     10      0

[2-3219.1904] eventpoll_pwq       1157   1216    208   19    1 : tunables    0    0    0 : slabdata     64     64      0

[2-3219.1904] eventpoll_epi       1156   1232    256   16    1 : tunables    0    0    0 : slabdata     77     77      0

[2-3219.1904] sgpool-128             2     10   3264   10    8 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1904] sgpool-64              2     18   1728   18    8 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1904] sgpool-32              2     17    960   17    4 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1904] sgpool-16              2     28    576   28    4 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1905] sgpool-8               2     21    384   21    2 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1905] scsi_data_buffer       0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1905] blkdev_queue          70     88   1448   22    8 : tunables    0    0    0 : slabdata      4      4      0

[2-3219.1905] blkdev_requests       84    147    384   21    2 : tunables    0    0    0 : slabdata      7      7      0

[2-3219.1905] blkdev_ioc           734    752    256   16    1 : tunables    0    0    0 : slabdata     47     47      0

[2-3219.1905] bio-0                  2    100    320   25    2 : tunables    0    0    0 : slabdata      4      4      0

[2-3219.1905] biovec-256             2     14   3264   10    8 : tunables    0    0    0 : slabdata      5      5      0

[2-3219.1905] biovec-128             0      0   1728   18    8 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1905] biovec-64              0     68    960   17    4 : tunables    0    0    0 : slabdata      4      4      0

[2-3219.1905] biovec-16              0     84    384   21    2 : tunables    0    0    0 : slabdata      4      4      0

[2-3219.1906] ksm_mm_slot          223    240    200   20    1 : tunables    0    0    0 : slabdata     12     12      0

[2-3219.1906] ksm_stable_node        0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1906] ksm_rmap_item     127089 128250    208   19    1 : tunables    0    0    0 : slabdata   6750   6750      0

[2-3219.1906] uid_cache             10     16    256   16    1 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1906] dmaengine-unmap-2      1     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1906] sock_inode_cache    1349   1425    640   25    4 : tunables    0    0    0 : slabdata     57     57      0

[2-3219.1906] skbuff_fclone_cache      0    112    576   28    4 : tunables    0    0    0 : slabdata      4      4      0

[2-3219.1907] skbuff_head_cache   2155   2247    384   21    2 : tunables    0    0    0 : slabdata    107    107      0

[2-3219.1907] file_lock_cache       55    100    320   25    2 : tunables    0    0    0 : slabdata      4      4      0

[2-3219.1907] file_lock_ctx        161    198    216   18    1 : tunables    0    0    0 : slabdata     11     11      0

[2-3219.1907] net_namespace          0      0   2688   12    8 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1907] shmem_inode_cache   2992   3048    664   24    4 : tunables    0    0    0 : slabdata    127    127      0

[2-3219.1907] pool_workqueue        26     32    512   16    2 : tunables    0    0    0 : slabdata      2      2      0

[2-3219.1907] proc_inode_cache      22    338    624   26    4 : tunables    0    0    0 : slabdata     13     13      0

[2-3219.1907] sigqueue               1     25    320   25    2 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1907] bdev_cache            94    105    768   21    4 : tunables    0    0    0 : slabdata      5      5      0

[2-3219.1908] kernfs_node_cache  29404  29472    256   16    1 : tunables    0    0    0 : slabdata   1842   1842      0

[2-3219.1908] mnt_cache            144    147    384   21    2 : tunables    0    0    0 : slabdata      7      7      0

[2-3219.1908] filp               31654  32508    448   18    2 : tunables    0    0    0 : slabdata   1806   1806      0

[2-3219.1909] inode_cache         5980   8721    592   27    4 : tunables    0

[read  2017-03-20 04:46:10.213208] 0    0 : slabdata    323    323      0

[2-3219.1911] dentry             13938  33456    328   24    2 : tunables    0    0    0 : slabdata   1394   1394      0

[2-3219.1911] names_cache            2     12   4288    7    8 : tunables    0    0    0 : slabdata      6      6      0

[2-3219.1911] iint_cache             0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1913] inode_smack        12900  28644    264   31    2 : tunables    0    0    0 : slabdata    924    924      0

[2-3219.1913] key_jar                2     25    320   25    2 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1913] buffer_head       162183 163217    240   17    1 : tunables    0    0    0 : slabdata   9601   9601      0

[2-3219.1913] nsproxy                3     20    200   20    1 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1914] vm_area_struct    102148 102960    272   30    2 : tunables    0    0    0 : slabdata   3432   3432      0

[2-3219.1914] mm_struct            223    230    704   23    4 : tunables    0    0    0 : slabdata     10     10      0

[2-3219.1914] fs_cache             226    240    256   16    1 : tunables    0    0    0 : slabdata     15     15      0

[2-3219.1914] files_cache          226    234    448   18    2 : tunables    0    0    0 : slabdata     13     13      0

[2-3219.1914] signal_cache         376    396    896   18    4 : tunables    0    0    0 : slabdata     22     22      0

[2-3219.1914] sighand_cache        376    382   1536   21    8 : tunables    0    0    0 : slabdata     20     20      0

[2-3219.1914] task_struct         1773   1822   1600   20    8 : tunables    0    0    0 : slabdata    101    101      0

[2-3219.1914] cred_jar             833    950    320   25    2 : tunables    0    0    0 : slabdata     38     38      0

[2-3219.1915] anon_vma_chain     45336  45467    208   19    1 : tunables    0    0    0 : slabdata   2393   2393      0

[2-3219.1915] anon_vma           35268  35428    232   17    1 : tunables    0    0    0 : slabdata   2084   2084      0

[2-3219.1915] pid                 1872   1920    256   16    1 : tunables    0    0    0 : slabdata    120    120      0

[2-3219.1916] radix_tree_node     6713   8279    480   17    2 : tunables    0    0    0 : slabdata    487    487      0

[2-3219.1916] ftrace_event_file    600    612    224   18    1 : tunables    0    0    0 : slabdata     34     34      0

[2-3219.1916] ftrace_event_field   2054   2071    208   19    1 : tunables    0    0    0 : slabdata    109    109      0

[2-32

[read  2017-03-20 04:46:11.214574] 19.1916] idr_layer_cache      437    468   1240   26    8 : tunables    0    0    0 : slabdata     18     18      0

[2-3219.1916] page->ptl          10040  10206    192   21    1 : tunables    0    0    0 : slabdata    486    486      0

[2-3219.1916] kmalloc-8192         135    139   8384    3    8 : tunables    0    0    0 : slabdata     47     47      0

[2-3219.1916] kmalloc-4096        3422   3423   4288    7    8 : tunables    0    0    0 : slabdata    891    891      0

[2-3219.1916] kmalloc-2048         145    196   2240   14    8 : tunables    0    0    0 : slabdata     14     14      0

[2-3219.1916] kmalloc-1024        2835   2872   1216   26    8 : tunables    0    0    0 : slabdata    114    114      0

[2-3219.1917] kmalloc-512         2521   2691    704   23    4 : tunables    0    0    0 : slabdata    117    117      0

[2-3219.1917] kmalloc-256          588    612    448   18    2 : tunables    0    0    0 : slabdata     34     34      0

[2-3219.1917] kmalloc-192         7553   7602    384   21    2 : tunables    0    0    0 : slabdata    362    362      0

[2-3219.1917] kmalloc-128        12477  13050    320   25    2 : tunables    0    0    0 : slabdata    522    522      0

[2-3219.1928] kmalloc-64        143648 171328    256   16    1 : tunables    0    0    0 : slabdata  10708  10708      0

[2-3219.1928] kmem_cache_node      169    176    256   16    1 : tunables    0    0    0 : slabdata     11     11      0

[2-3219.1928] kmem_cache           169    175    320   25    2 : tunables    0    0    0 : slabdata      7      7      0

** 37 printk messages dropped ** [2-3219.1999] bb-mini** 325 printk messages dropped **

 

Analysis

A)

[2-3219.1870] vd-net-config invoked oom-killer: gfp_mask=0x80d0, order=0, oom_score_adj=-900

[2-3219.1870] CPU: 2 PID: 467 Comm: vd-net-config Tainted: PO 4.1.10 #1

[2-3219.1870] SCHED_NORMAL (p:120, static_p:120, normal_p:120, rt_p:0)

Here, vd-net-config – process vd-net-config asks for memory allocation

gfp_mask – tells the requested allocation zone (DMA(NA for VD), NORMAL, HIGH)

order – size of requested page

 

B)

[2-3219.1873] Mem-Info:

[2-3219.1875] active_anon:148141 inactive_anon:52931 isolated_anon:0

[2-3219.1875]  active_file:29121 inactive_file:30395 isolated_file:50

[2-3219.1875]  unevictable:2 dirty:17 writeback:0 unstable:0

[2-3219.1875]  slab_reclaimable:15062 slab_unreclaimable:54733

[2-3219.1875]  mapped:24034 shmem:7265 pagetables:10040 bounce:0

[2-3219.1875]  free:20647 free_pcp:0 free_cma:16100

Here, anon and file means that:

A page is said to be mapped if it maps a portion of a file. For instance, all pages in the User Mode address spaces belonging to file memory mappings are mapped, as well as any other page included in the page cache. In almost all cases, mapped pages are syncable: in order to reclaim the page frame, the kernel must check whether the page is dirty and, if necessary, write the page contents in the corresponding disk file (like files opened by open() system call etc).

Conversely, a page is said to be anonymous if it belongs to an anonymous memory region of a process (for instance, all pages in the User Mode heap or stack of a process are anonymous). In order to reclaim the page frame, the kernel must save the page contents in a dedicated disk partition or disk file called “swap area” therefore, all anonymous pages are swappable.

 

C)

active_anon – (Active(anon) in /proc/meminfo) number of pages of Anonymous memory that has been used more recently and usually not swapped out. This is = (Normal active_anon + HighMem active_anon) / 4

inactive_anon – (Inactive(anon) in /proc/meminfo) number of pages of Anonymous memory that has not been used recently and can be swapped out.  This is = (Normal inactive_anon + HighMem inactive_anon) / 4

isolated_anon – More info is required(todo)…  This is = (Normal isolated(anon) + HighMem isolated(anon)) / 4

active_file – (Active(file) in /proc/meminfo) number of pages of Pagecache memory that has been used more recently and usually not reclaimed until needed. This is = (Normal active_file + HighMem active_file) / 4

inactive_file – (Inactive(file) in /proc/meminfo) number of pages of Pagecache memory that can be reclaimed without huge performance impact.  This is = (Normal inactive_file + HighMem inactive_file) / 4

isolated_file – More info is required(todo)… This is = (Normal isolated(file) + HighMem isolated(file)) / 4

unevictable – (Unevictable in /proc/meminfo) number of pages of Unevictable pages can’t be swapped out for a variety of reasons. This is = (Normal unevictable + HighMem unevictable) / 4

dirty – (Dirty in /proc/meminfo) number of pages of Memory waiting to be written back to disk.  This is = (Normal dirty + HighMem dirty) / 4

writeback – (Writeback in /proc/meminfo) number of pages of Memory which is actively being written back to disk.  This is = (Normal writeback + HighMem writeback) / 4

unstable – (NFS_Unstable in /proc/meminfo) number of pages of NFS pages sent to the server but not yet committed to the stable storage.  This is = (Normal unstable + HighMem unstable) / 4

slab_reclaimable – (SReclaimable in /proc/meminfo) number of pages of The part of Slab that can be reclaimed, such as caches. slab is the amount of memory being used for kernel mallocs. So, it is in-kernel data structures cache.  This is = (Normal slab_reclaimable + HighMem slab_reclaimable) / 4

slab_unreclaimable – (SUnreclaim in /proc/meminfo) number of pages of The part of Slab that cannot be reclaimed even when lacking memory.   This is = (Normal slab_unreclaimable + HighMem slab_unreclaimable) / 4

mapped – (Mapped in /proc/meminfo) number of pages of The memory used for files that have been mmaped, such as libraries..   This is = (Normal mapped + HighMem mapped) / 4

shmem – (Shmem in /proc/meminfo) number of pages of The total amount of memory used by shared memory (shmem) and tmpfs. .   This is = (Normal shmem + HighMem shmem) / 4

pagetables – (PageTables in /proc/meminfo) number of pages of  The total amount of memory dedicated to the lowest page table level. This can increase to a high value if a lot of processes are attached to the same shared memory segment..   This is = (Normal pagetables + HighMem pagetables) / 4

free – (MemFree in /proc/meminfo) number of pages of The amount of physical RAM left unused by the system. This is = (Normal free + HighMem free) / 4

free_pcp – More info is required(todo)..   This is = (Normal free_pcp + HighMem free_pcp) / 4

free_cma – More info is required(todo)..   This is = (Normal free_cma + HighMem free_cma) / 4

 

D)

[2-3219.1875] Normal free:67368kB min:3032kB low:3788kB high:4548kB active_anon:0kB inactive_anon:24kB active_file:148kB inactive_file:160kB unevictable:0kB isolated(anon):0kB isolated(file):200kB present:622592kB managed:576420kB mlocked:0kB dirty:12kB writeback:4kB mapped:316kB shmem:0kB slab_reclaimable:60248kB slab_unreclaimable:218932kB kernel_stack:14176kB pagetables:40160kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:64400kB writeback_tmp:0kB pages_scanned:6352 all_unreclaimable? yes

E)

[2-3219.1875] lowmem_reserve[]: 0 94 94

[2-3219.1875] HighMem free:15220kB min:512kB low:2540kB high:4568kB active_anon:592564kB inactive_anon:211700kB active_file:116336kB inactive_file:121420kB unevictable:8kB isolated(anon):0kB isolated(file):0kB present:1542144kB managed:1542140kB mlocked:8kB dirty:56kB writeback:0kB mapped:95820kB shmem:29060kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no

[2-3219.1875] lowmem_reserve[]: 0 0 0

Here, Zone – Normal zone memory

free – free memory in this zone

min:3032kB low:3788kB high:4548kB – These are watermarks. oom killer is invoked when memory requested is for this zone and available memory has gone less than min(3032kB). When it breaches low(3788kB) but more than min(3032kB), kswapd is triggered until available memory reaches high(4548kB)

present – tells us how large the zone is.present_pages is physical pages existing within the zone.

managed – managed_pages is present pages managed by the buddy system. More info is given below:

spanned_pages is the total pages spanned by the zone, including holes, which is calculated as:

spanned_pages = zone_end_pfn – zone_start_pfn;

present_pages is physical pages existing within the zone, which is calculated as:

present_pages = spanned_pages – absent_pages(pages in holes);

managed_pages is present pages managed by the buddy system, which is calculated as (reserved_pages includes pages allocated by the bootmem allocator):

managed_pages = present_pages – reserved_pages;

So present_pages may be used by memory hotplug or memory power management logic to figure out unmanaged pages by checking (present_pages – managed_pages). And managed_pages should be used by page allocator and vm scanner to calculate all kinds of watermarks and thresholds.

mlocked – The total amount of memory, in kilobytes, that is not evictable because it is locked into memory by user programs.

kernel_stack – More info is required(todo)…

bounce – The amount of memory, in kilobytes, used for the block device “bounce buffers”.

local_pcp – More info is required(todo)…

writeback_tmp – The amount of memory, in kilobytes, used by FUSE for temporary writeback buffers.

pages_scanned – This counts number of pages scanned since the last page free event in the allocator. This was used primarily to measure the reclaimability of zones and nodes, and determine when reclaim should give up on them.

all_unreclaimable – field is set to 1(yes) if the pageout daemon scans through all the pages in the zone twice and still fails to free enough pages.

lowmem_reserve – This is the ‘lowmem reservation’ level for this zone. When the kernel is handling an allocation that started in another zone (High on this machine), the Normal zone must have at least a certain number of pages free in order to be a fallback candidate for the allocation (sort of, it actually gets quite complicated). Here the numbers are 0 pages free for Normal allocations, 94 pages free for High allocations. Since the Normal zone has less than 94 pages, it’s perhaps not too surprising that it’s mostly free right now.

results with 256, 32 in the lowmem_reserve sysctl:

1G machine -> (16M dma, 800M-16M normal, 1G-800M high)

1G machine -> (16M dma, 784M normal, 224M high)

NORMAL allocation will leave 784M/256 of ram reserved in the ZONE_DMA

HIGHMEM allocation will leave 224M/32 of ram reserved in ZONE_NORMAL

HIGHMEM allocation will (224M+784M)/256 of ram reserved in ZONE_DMA

Example: let’s assume a 1 GiB node with 768 MiB in zone Normal and 256 MiB in zone HighMem (assume no zone DMA). Let’s assume the default highmem reserve “ratio” (divisor) of 32. And let’s assume the typical 4 KiB page size. Now we can calculate the reserve area for zone Normal:

1.Sum of “higher” zones than zone Normal (just HighMem): 256 MiB = (1024 KiB / 1 MiB) * (1 page / 4 KiB) = 65536 pages

2.Area reserved in zone Normal for this node: 65536 pages / 32 = 2048 pages = 8 MiB.

So when calculating the reserve pages for a given zone in a node, you take the sum of pages in that node in zones higher than that one, divide it by the provided divisor, and that’s how many pages you’re reserving for that zone.

 

[2-3219.1875] VmallocUsed:      221008 kB [    8192 kB vm_map_ram,   187984 kB ioremap,    21972 kB vmalloc,      212 kB vmap,     2656 kB usermap,     8196 kB vpages,     8204 kB overlapped]

Here, VmallocUsed – The total amount of memory, in kilobytes, of used virtual address space.

vm_map_ram – More info is required(todo)…

ioremap – More info is required(todo)…

vmalloc – More info is required(todo)…

vmap – More info is required(todo)…

usermap – More info is required(todo)…

vpages – More info is required(todo)…

overlapped – More info is required(todo)…

 

G)

[2-3219.1875] Normal: 75*4kB (UERC) 53*8kB (URC) 84*16kB (UEMC) 32*32kB (RC) 24*64kB (C) 15*128kB (RC) 8*256kB (RC) 1*512kB (R) 3*1024kB (RC) 1*2048kB (C) 13*4096kB (C) = 67476kB

[2-3219.1876] HighMem: 2547*4kB (UMR) 208*8kB (UMR) 28*16kB (UM) 11*32kB (UMR) 25*64kB (M) 8*128kB (UMR) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15276kB

Here,

static const char types[MIGRATE_TYPES] = {

[MIGRATE_UNMOVABLE]     = ‘U’,

[MIGRATE_RECLAIMABLE]   = ‘E’,

[MIGRATE_MOVABLE]       = ‘M’,

[MIGRATE_RESERVE]       = ‘R’,

#ifdef CONFIG_CMA

[MIGRATE_CMA]           = ‘C’,

#endif

#ifdef CONFIG_MEMORY_ISOLATION

[MIGRATE_ISOLATE]       = ‘I’,

#endif

};

Page migration allows the moving of the physical location of pages between nodes in a numa system while the process is running. This means that the virtual addresses that the process sees do not change. However, the system rearranges the physical location of those pages. The main intend of page migration is to reduce the latency of memory access by moving pages near to the processor where the process accessing that memory is running (https://www.kernel.org/doc/Documentation/vm/page_migration).

When requesting memory from the buddy allocator, one provides a gfp_mask. Among other things, it specifies the “migrate type” of the requested page(s). One of the migrate types is MIGRATE_MOVABLE. The idea behind it is that data from a movable page can be migrated (or moved, hence the name), which works well for disk caches, process pages, etc.

To keep pages with the same migrate type together, the buddy allocator groups pages into “pageblocks,” each having a migrate type assigned to it. The allocator then tries to allocate pages from pageblocks with a type corresponding to the request. If that’s not possible, however, it will take pages from different pageblocks and may even change a pageblock’s migrate type. This means that a non-movable page can be allocated from a MIGRATE_MOVABLE pageblock which can also result in that pageblock changing its migrate type. This is undesirable for CMA, so it introduces a MIGRATE_CMA type which has one important property: only movable pages can be allocated from a MIGRATE_CMA pageblock (https://lwn.net/Articles/486301/).

The pages with C (MIGRATE_CMA) will not be considered for allocation. Only movable pages can be allocated from MIGRATE_CMA pageblocks and page allocator never implicitly change migration type of MIGRATE_CMA pageblock.

More info is required(todo)…

 

H)

[2-3219.1876] 71761 total pagecache pages

[2-3219.1876] 4915 pages in swap cache

[2-3219.1876] Swap cache stats: add 163067, delete 158152, find 9884/23756

[2-3219.1876] Free swap  = 607248kB

[2-3219.1876] Total swap = 1048572kB

Here, total pagecache pages – NR_FILE_PAGES, More info is required(todo)…

pages in swap cache = add 163067 – delete 158152 = 4915

The Swap Cache:When swapping pages out to the swap files, Linux avoids writing pages if it does not have to. There are times when a page is both in a swap file and in physical memory. This happens when a page that was swapped out of memory was then brought back into memory when it was again accessed by a process. So long as the page in memory is not written to, the copy in the swap file remains valid.

Linux uses the swap cache to track these pages. The swap cache is a list of page table entries, one per physical page in the system. This is a page table entry for a swapped out page and describes which swap file the page is being held in together with its location in the swap file. If a swap cache entry is non-zero, it represents a page which is being held in a swap file that has not been modified. If the page is subsequently modified (by being written to), its entry is removed from the swap cache.

When Linux needs to swap a physical page out to a swap file it consults the swap cache and, if there is a valid entry for this page, it does not need to write the page out to the swap file. This is because the page in memory has not been modified since it was last read from the swap file.

The entries in the swap cache are page table entries for swapped out pages. They are marked as invalid but contain information which allow Linux to find the right swap file and the right page within that swap file. (http://www.linux-tutorial.info/modules.php?name=MContent&pageid=314)

Swap cache stats: add 163067 (total pages added in swap cache), delete 158152 (total pages deleted from swap cache), find 9884/23756 (success in finding page in swap cache/total searches)

Free swap – (SwapFree in /proc/meminfo) The total amount of swap free, in kilobytes.

Total swap – (SwapTotal in /proc/meminfo) The total amount of swap available, in kilobytes.

 

I)

[2-3219.1876] 541184 pages RAM

[2-3219.1876] 385536 pages HighMem/MovableOnly

[2-3219.1876] 4294962456 pages reserved

[2-3219.1876] 16384 pages cma reserved

Here, pages RAM –  This is = (Normal present + HighMem present) / 4

pages HighMem/MovableOnly – This is = (HighMem present) / 4

pages reserved – This is = [ ((Normal present – Normal managed) + (HighMem present – HighMem managed)) / 4 ] – (pages cma reserved below)

pages cma reserved – More info is required(todo)…

 

J)

[2-3219.1876] [ pid ]   uid  tgid  ppid total_vm      rss  rss_max   swap cpu oom_score_adj name

[2-3219.1876] [  174]     0   174     1   222960    11452    23476   3232   2     -1000 enlightenment

[2-3219.1877] [  180]     0   180     1     3296      100      960    104   2     -1000 cynara-launcher

[2-3219.1877] [  182]     0   182     1   104604     1296     5112    928   3         0 systemd-boot-he

[2-3219.1877] [  184]     0   184     1     3292       96      924    104   0     -1000 security-manage

[2-3219.1877] [  192]     0   192     1    24108      152      864    156   2         0 predevattr

[2-3219.1877] [  194]     0   194   184    15464     1500     3844    296   3     -1000 security-manage

[2-3219.1877] [  198]   401   198   180     6016     1248     3420    236   1     -1000 cynara

[2-3219.1877] [  207]     0   207     1    12548      564     3072    452   0     -1000 smack-logger

[2-3219.1877] [  252]     0   252     1    10736      392     2644    304   3         0 kfactoryd

[2-3219.1877] [  276]     0   276     1     8668      308     2196    208   3     -1000 buxton2d

[2-3219.1877] [  281]   375   281   276    15380     3040     4952    528   1     -1000 buxton2d

[2-3219.1877] [  302]  5001   302     1    33300      956     3712    648   2     -1000 launchpad-proce

*

*

*

[2-3219.1892] [19319]     0 19319 19313   106068     9812    12160      0   2     -1000 picl

[2-3219.1892] [19401]     0 19401 19313    10120     2168     2512      0   0     -1000 shop_contents_d

[2-3219.1892] [19403]     0 19403 19401     7596     1792     2084      0   1     -1000 curl

[2-3219.1892] * Sum of rss    : 1239568 (kB)

[2-3219.1892] * Sum of maxrss : 3555856 (kB)

[2-3219.1892] * Sum of swap   : 421956 (kB)

Here, these are the prints of dump_tasks

Sum of rss – sum of rss of all the processes

Sum of maxrss – sum of rss_max of all the processes

Sum of swap – sum of swap of all the processes. It is approx equal to the (Total swap – Free swap) (above)

 

K)

[2-3219.1892] slabinfo – version: 2.1

[2-3219.1892] # name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>

[2-3219.1892] pid_2                  5     16    256   16    1 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1892] ecryptfs_key_tfm_cache      0      0    320   25    2 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1893] ecryptfs_global_auth_tok_cache      2     17    960   17    4 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1893] ecryptfs_key_sig_cache      0      0    224   18    1 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1893] ecryptfs_key_record_cache      0      0    704   23    4 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1893] ecryptfs_xattr_cache      0      0   4288    7    8 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1893] ecryptfs_headers       0      0   4288    7    8 : tunables    0    0    0 : slabdata      0      0      0

[2-3219.1893] ecryptfs_sb_cache      1     18    896   18    4 : tunables    0    0    0 : slabdata      1      1      0

[2-3219.1893] ecryptfs_inode_cache      1     16   1024   16    4 : tunables    0    0    0 : slabdata      1      1      0

 

L)

[1-3222.2593] Out of memory: Kill process 6801 (ignitionLaunch) score 1008 or sacrifice child

[1-3222.2593] Killed process 6801 (ignitionLaunch) total-vm:339908kB, anon-rss:56336kB, file-rss:3224kB

[1-3222.2593] [kpi_fault_info] [k] 2 -> p:0x00000000 l:0x00000000 th:OOM pro:OOM ty:OOM pgid:6801

Here, process 6801 (ignitionLaunch) is killed by oom killer

total-vm – the size of the virtual memory that a process uses is listed as total-vm. Part of it is really mapped into the RAM itself (allocated and used). This is RSS.

anon-rss – Part of the RSS is allocated in real memory blocks (other than mapped into a file or device). This is anonymous memory (anon-rss).

file-rss – there is also RSS memory blocks that are mapped into devices and files (file-rss).

 

M)

So, if you open a huge file in vim, the file-rss would be high, on the other size, if you malloc() a lot of memory and really use it, your anon-rss would be high also. On the other side, if you allocate a lot of space (with malloc()), but nevers use it, the total-vm would be higher, but no real memory would be used (due to the memory overcommit), so, the rss values would be low.(http://stackoverflow.com/questions/18845857/what-does-anon-rss-and-total-vm-mean)

 

N)

This can be verified as below:

RSS of ignitionLaunch in dump_tasks() list above = anon-rss:56336kB + file-rss:3224kB (approx. very close)

************************************************************

Why do oom occurs ? Give all the scenarios:

  1. check the zone Z and order of allocation O requested.
  2. If (free – free_cma) < min, oom occurs. This oom is occured because free memory has gone lower than min.
  3. In the second type, oom occurs due to fragmentation. The pages should be empty in the order higher or equal to that of requested order O in zone Z. Moreover, the memory will not be allocated even if the pages are available but only of C type (MIGRATE_CMA).
  4. Third type : There is more to allocating from an order than just checking the amount of free memory available for that order or higher. The kernel effectively subtracts memory from all lower orders from the total free line and then performs the min watermark check on what is left (http://serverfault.com/questions/564068/linux-oom-situation).
  5. Fourth type: when free memory is more than min watermark and less than low watermark. In this case, kswapd gets triggered and tries to swap pages until that value gets back up above the high watermark value. In case “Free swap = 0kB” above, So basically kswapd triggered, but swap was full, so it couldn’t do anything so the only option was to start killing things until it could get pages_free back up (http://serverfault.com/questions/362506/oom-killer-killing-things-with-plenty-of-free-ram).
  6. Fifty type (https://utcc.utoronto.ca/~cks/space/blog/linux/WhyPageAllocFailure):

Example-

Node 0 DMA32 free:12560kB min:7076kB low:8844kB high:10612kB […]

Node 0 DMA32: 2360*4kB 80*8kB 21*16kB 7*32kB 4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 12560kB

On first glance it looks like everything should be fine here, because there is a free 64 Kb chunk and the zone has more free memory than any of the low water marks (especially min), but it turns out that the kernel does something tricky for higher-order allocations (allocations that aren’t for a single 4 Kb page). To simplify and generalize, the kernel has decided that when checking free memory limits, if you’re asking for a higher-order page the free memory in lower-order pages shouldn’t count towards the amount of memory it considers free (presumably because such ‘free memory’ can’t satisfy your request). At the same time it has to reduce the minimum amount of free memory required to avoid absurd results.

(One little thing is that the kernel check is made based on what the free memory would be after your allocation is made. This probably won’t matter for small requests but might matter if you ask for an order 10 allocation of four megabytes.)

So for higher-order allocations only memory available at that order and higher counts, and the starting low water mark is divided by two for every order above 0, ie for an order 4 request like ours the water marks wind up divided by 16 (which conveniently is 2^order). In theory in our situation this means that the kernel would consider there to be 1920 Kb free in DMA32 (well, 1856 Kb after we take off our allocation) instead of 12560 Kb and the minimum low water mark would be 442 Kb instead of 7076 Kb. This still looks like our allocation request should pass muster.

However, the kernel doesn’t actually implement the check this way in a single computation. Instead it does it iteratively, using a loop that is done for each order up to (but not including) the order of your request. In pseudo-code:

for each order starting with 0 up to (our order – 1):

free memory -= free memory for the current order

minimum memory = minimum memory / 2

if free memory <= minimum memory:

return failure

The problem is that this iterative approach causes an early failure if a significant amount of the free memory in a zone is in very low order pages, because you can lose a lot of free memory while the current minimum memory requirement only drops by a bit (well, by half). In our situation, much of the free memory in DMA32 is in order 0 pages so the first pass through the loop gives us a new free memory of 3056 Kbytes (12560 Kb minus 9440 Kb of order-0 pages and our 64 Kb request) but a minimum memory requirement of 3538 Kb (the initial 7076 Kb divided by two) and the code immediately declares that there is not enough memory in this zone.

  1. Sixth Type: More conclusions…(https://linuxraj.wordpress.com/2015/11/18/what-are-out-of-memory-events/)

Node 0 Normal free:17320kB min:16956kB low:21192kB high:25432kB active_anon:455352kB inactive_anon:455504kB active_file:140kB inactive_file:404kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1034240kB mlocked:0kB dirty:0kB writeback:0kB mapped:240kB shmem:4kB slab_reclaimable:9552kB slab_unreclaimable:30024kB kernel_stack:2320kB pagetables:16104kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no

active_anon:671109 inactive_anon:254487 isolated_anon:0

active_file:34 inactive_file:147 isolated_file:0

unevictable:0 dirty:0 writeback:0 unstable:0

free:22023 slab_reclaimable:3076 slab_unreclaimable:7939

mapped:87 shmem:3 pagetables:6383 bounce:0

Free swap  = 0kB

Total swap = 4194296kB

7(a) Here, The Normal zone has ~1GB (present:1034240kB) of memory and there is ~900MB of anon (application) pages (active_anon:455352kB + inactive_anon:455504kB) in use and only 500KB of file (cached) pages (active_file:140kB + inactive_file:404kB). The system has a total of 671109+254487 application pages but only 34+147 file pages so the cache has been almost completely reclaimed.

7(b) There is no free swap left from the 4GB of total swap space (Free swap  = 0kB Total swap = 4194296kB).

  1. Total RAM:

Active (in /proc/meminfo) = Active(anon) (in /proc/meminfo) + Active(file) (in /proc/meminfo) = active_anon (in oom log) + active_file (in oom log)

Inactive (in /proc/meminfo) = Inactive(anon) (in /proc/meminfo) + Inactive(file) (in /proc/meminfo) = inactive_anon (in oom log) + inactive_file (in oom log)

Slab (in /proc/meminfo) = SReclaimable (in /proc/meminfo) + SUnreclaim (in /proc/meminfo) = slab_reclaimable (in oom log) + slab_unreclaimable (in oom log)

(Buffers (in /proc/meminfo) + Cached (in /proc/meminfo) + SwapCached (in /proc/meminfo)) + AnonPages (in /proc/meminfo) = Active (in /proc/meminfo) + Inactive (in /proc/meminfo)

Here, Memory currently allocated to hold data backed by files. This included pages currently memory-mapped as well as pages still cached from recent accesses. The distinction between ‘buffers’ and ‘cached’ is becoming more and more pointless; as a rule of thumb you can say file meta-data (e.g. inodes, directories, indirect lists, …) count as ‘buffers’, while the actual file ‘payload’ data counts as ‘cached’ (http://tiebing.blogspot.kr/2009/10/understand-linux-procmeminfo.html).

MemTotal (in /proc/meminfo) = MemFree(in /proc/meminfo) + (Buffers + Cached + SwapCached) + AnonPages + (Slab + PageTables (in /proc/meminfo) + KernelStack (in /proc/meminfo))

MemTotal (in /proc/meminfo) = MemFree(of all zones of oom log) + (Buffers + Cached + SwapCached) + AnonPages + ((slab_reclaimable + slab_unreclaimable) + pagetables (in oom log) + kernel_stack (in oom log))

MemTotal (in /proc/meminfo) = MemFree(of all zones of oom log) + (Active + Inactive) + (Slab + pagetables + kernel_stack)

MemTotal (in /proc/meminfo) = MemFree(of all zones of oom log) + ((“active_anon” + “active_file”) + (“inactive_anon” + “inactive_file”)) + ((slab_reclaimable + slab_unreclaimable) + pagetables + kernel_stack)

(https://blog.famzah.net/tag/linux-slab/)

More explanation:

What is the difference between Buffers and Cached columns in /proc/meminfo output?

Short answer: Cached is the size of the page cache. Buffers is the size of in-memory block I/O buffers. Cached matters; Buffers is largely irrelevant.

Long answer: Cached is the size of the Linux page cache, minus the memory in the swap cache, which is represented by SwapCached (thus the total page cache size is Cached + SwapCached). Linux performs all file I/O through the page cache. Writes are implemented as simply marking as dirty the corresponding pages in the page cache; the flusher threads then periodically write back to disk any dirty pages. Reads are implemented by returning the data from the page cache; if the data is not yet in the cache, it is first populated. On a modern Linux system, Cached can easily be several gigabytes. It will shrink only in response to memory pressure. The system will purge the page cache along with swapping data out to disk to make available more memory as needed.

Buffers are in-memory block I/O buffers. They are relatively short-lived. Prior to Linux kernel version 2.4, Linux had separate page and buffer caches. Since 2.4, the page and buffer cache are unified and Buffers is raw disk blocks not represented in the page cache—i.e., not file data. The Buffers metric is thus of minimal importance. On most systems, Buffers is often only tens of megabytes.(https://www.quora.com/What-is-the-difference-between-Buffers-and-Cached-columns-in-proc-meminfo-output)

 

The page cache caches pages of files to optimize file I/O. The buffer cache caches disk blocks to optimize block I/O.

Prior to Linux kernel version 2.4, the two caches were distinct: Files were in the page cache, disk blocks were in the buffer cache. Given that most files are represented by a filesystem on a disk, data was represented twice, once in each of the caches. Many Unix systems follow a similar pattern.

This is simple to implement, but with an obvious inelegance and inefficiency. Starting with Linux kernel version 2.4, the contents of the two caches were unified. The VM subsystem now drives I/O and it does so out of the page cache. If cached data has both a file and a block representation—as most data does—the buffer cache will simply point into the page cache; thus only one instance of the data is cached in memory. The page cache is what you picture when you think of a disk cache: It caches file data from a disk to make subsequent I/O faster.

The buffer cache remains, however, as the kernel still needs to perform block I/O in terms of blocks, not pages. As most blocks represent file data, most of the buffer cache is represented by the page cache. But a small amount of block data isn’t file backed—metadata and raw block I/O for example—and thus is solely represented by the buffer cache.(https://www.quora.com/What-is-the-major-difference-between-the-buffer-cache-and-the-page-cache)

 

  1. Total user space Memory = ((“active_anon” + “active_file”) + (“inactive_anon” + “inactive_file”))
  2. Total kernel space Memory = ((slab_reclaimable + slab_unreclaimable) + pagetables + kernel_stack)
  3. Total SWAP = SwapTotal (in /proc/meminfo) = Total swap (in oom log)
  4. Free Swap = SwapFree (in /proc/meminfo) = Free swap (in oom log)
  5. Total GEM / MALI = More info is required(todo)…
  6. Total Available memory:

Total free memory F = Normal free + HighMem free

wmark_low of both zones W = Normal low + HighMem low

pagecache P = active_file + inactive_file

P -= min(P / 2, wmark_low)

Total system Available memory (of both the zones ie Normal and High zones) A = F – W

A = A + P

A = A + slab_reclaimable – min(slab_reclaimable / 2, wmark_low)

A = A – free_cma

 

 

Total free memory F = Normal free + HighMem free = 69524

wmark_low of both zones W = Normal low + HighMem low = 5280

pagecache P = active_file + inactive_file = 46136

P -= min(P / 2, wmark_low) = 40856

Total system Available memory (of both the zones ie Normal and High zones) A = F – W = 64244

A = A + P = 105100

A = A + slab_reclaimable – min(slab_reclaimable / 2, wmark_low) = 216656

A = A – free_cma = 153228

 

Total free memory F = Normal free + HighMem free = 69816

wmark_low of both zones W = Normal low + HighMem low = 5280

pagecache P = active_file + inactive_file = 46368

P -= min(P / 2, wmark_low) = 41088

Total system Available memory (of both the zones ie Normal and High zones) A = F – W = 64536

A = A + P = 105624

A = A + slab_reclaimable – min(slab_reclaimable / 2, wmark_low) = 217280

A = A – free_cma = 153852