[linux-rpi] OOM killer issues (armv7h)

Discussion about U-Boot and the kernel.

Re: [linux-rpi] OOM killer issues (armv7h)

Postby graysky » Sat Mar 18, 2023 11:28 am

MOD NOTE - merged
graysky
Developer
 
Posts: 1727
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000

Re: [linux-rpi] OOM killer issues (armv7h)

Postby bxabi » Sun Mar 19, 2023 9:36 am

I have the same problem, after 2 years of working fine now rsync triggers the OOM memory killer.

$this->bbcode_second_pass_code('', ' rsync invoked oom-killer: gfp_mask=0xcd0(GFP_KERNEL|__GFP_RECLAIMABLE), order=0, oom_score_adj=0
Mar 19 04:08:56 rasp4 kernel: CPU: 1 PID: 1164 Comm: rsync Tainted: G C 6.1.19-2-rpi-ARCH #1
Mar 19 04:08:56 rasp4 kernel: Hardware name: BCM2711
Mar 19 04:08:56 rasp4 kernel: unwind_backtrace from show_stack+0x18/0x1c
Mar 19 04:08:56 rasp4 kernel: show_stack from dump_stack_lvl+0x90/0xac
Mar 19 04:08:56 rasp4 kernel: dump_stack_lvl from dump_header+0x54/0x1fc
Mar 19 04:08:56 rasp4 kernel: dump_header from oom_kill_process+0x23c/0x248
Mar 19 04:08:56 rasp4 kernel: oom_kill_process from out_of_memory+0x218/0x34c
Mar 19 04:08:56 rasp4 kernel: out_of_memory from __alloc_pages+0xa98/0x1044
Mar 19 04:08:56 rasp4 kernel: __alloc_pages from new_slab+0x384/0x43c
Mar 19 04:08:56 rasp4 kernel: new_slab from ___slab_alloc+0x3e8/0xa0c
Mar 19 04:08:56 rasp4 kernel: ___slab_alloc from kmem_cache_alloc_lru+0x4fc/0x640
Mar 19 04:08:56 rasp4 kernel: kmem_cache_alloc_lru from __d_alloc+0x2c/0x1bc
Mar 19 04:08:56 rasp4 kernel: __d_alloc from d_alloc+0x18/0x74
Mar 19 04:08:56 rasp4 kernel: d_alloc from d_alloc_parallel+0x50/0x3b8
Mar 19 04:08:56 rasp4 kernel: d_alloc_parallel from __lookup_slow+0x60/0x138
Mar 19 04:08:56 rasp4 kernel: __lookup_slow from walk_component+0xf4/0x164
Mar 19 04:08:56 rasp4 kernel: walk_component from path_lookupat+0x7c/0x1a4
Mar 19 04:08:56 rasp4 kernel: path_lookupat from filename_lookup+0xc0/0x190
Mar 19 04:08:56 rasp4 kernel: filename_lookup from vfs_statx+0x7c/0x168
Mar 19 04:08:56 rasp4 kernel: vfs_statx from do_statx+0x70/0xb0
')
bxabi
 
Posts: 1
Joined: Sun Mar 19, 2023 9:33 am

Re: [linux-rpi] OOM killer issues (armv7h)

Postby graysky » Sun Mar 19, 2023 10:20 am

Can someone open an issue upstream.
graysky
Developer
 
Posts: 1727
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000

OOM Killer gone rogue in RPi 4

Postby TheDcoder » Sun Mar 19, 2023 6:23 pm

Hi,

The OOM Killer has gone rouge in my system since a couple of weeks, it keeps killing processes at random even though there is plenty of free RAM (> 85% always).

It has especially taken a liking to killing qBittorrent (nox). If I keep the system running for long enough, it will even kill systemd and make the system useless essentially.

Here's a typical kill session:

$this->bbcode_second_pass_code('', '
Mar 19 23:16:10 rpi kernel: bash invoked oom-killer: gfp_mask=0x400dc0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), order=0, oom_score_adj=0
Mar 19 23:16:10 rpi kernel: CPU: 3 PID: 741 Comm: bash Tainted: G C 6.1.19-2-rpi-ARCH #1
Mar 19 23:16:10 rpi kernel: Hardware name: BCM2711
Mar 19 23:16:10 rpi kernel: unwind_backtrace from show_stack+0x18/0x1c
Mar 19 23:16:10 rpi kernel: show_stack from dump_stack_lvl+0x90/0xac
Mar 19 23:16:10 rpi kernel: dump_stack_lvl from dump_header+0x54/0x1fc
Mar 19 23:16:10 rpi kernel: dump_header from oom_kill_process+0x23c/0x248
Mar 19 23:16:10 rpi kernel: oom_kill_process from out_of_memory+0x218/0x34c
Mar 19 23:16:10 rpi kernel: out_of_memory from __alloc_pages+0xa98/0x1044
Mar 19 23:16:10 rpi kernel: __alloc_pages from __pmd_alloc+0x3c/0x1d8
Mar 19 23:16:10 rpi kernel: __pmd_alloc from pgd_alloc+0x2e0/0x32c
Mar 19 23:16:10 rpi kernel: pgd_alloc from mm_init+0x100/0x1cc
Mar 19 23:16:10 rpi kernel: mm_init from dup_mm+0x60/0x5a4
Mar 19 23:16:10 rpi kernel: dup_mm from copy_process+0xda0/0x164c
Mar 19 23:16:10 rpi kernel: copy_process from kernel_clone+0xac/0x3a8
Mar 19 23:16:10 rpi kernel: kernel_clone from sys_clone+0x78/0x9c
Mar 19 23:16:10 rpi kernel: sys_clone from ret_fast_syscall+0x0/0x1c
Mar 19 23:16:10 rpi kernel: Exception stack(0xf0c31fa8 to 0xf0c31ff0)
Mar 19 23:16:10 rpi kernel: 1fa0: b6fa8088 00000001 01200011 00000000 00000000 00000000
Mar 19 23:16:10 rpi kernel: 1fc0: b6fa8088 00000001 b6ebae58 00000078 befc624c 0057d2bc befc61cc 005a44e0
Mar 19 23:16:10 rpi kernel: 1fe0: b6fa85a0 befc6058 b6ded260 b6ded684
Mar 19 23:16:10 rpi kernel: Mem-Info:
Mar 19 23:16:10 rpi kernel: active_anon:104352 inactive_anon:245 isolated_anon:0
active_file:39467 inactive_file:48297 isolated_file:0
unevictable:0 dirty:3095 writeback:0
slab_reclaimable:3924 slab_unreclaimable:7175
mapped:31985 shmem:265 pagetables:1022
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:764189 free_pcp:663 free_cma:124679
Mar 19 23:16:10 rpi kernel: Node 0 active_anon:417408kB inactive_anon:980kB active_file:157868kB inactive_file:193188kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:127940kB dirty:12380kB writeback:0kB shmem:1060kB writeback_tmp:0kB kernel_stack:1984kB pagetables:4088kB sec_pagetables:0kB all_unreclaimable? yes
Mar 19 23:16:10 rpi kernel: DMA free:502580kB boost:7272kB min:10504kB low:11312kB high:12120kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:712kB inactive_file:84296kB unevictable:0kB writepending:11944kB present:786432kB managed:663844kB mlocked:0kB bounce:0kB free_pcp:2652kB local_pcp:520kB free_cma:498716kB
Mar 19 23:16:10 rpi kernel: lowmem_reserve[]: 0 0 3188 3188
Mar 19 23:16:10 rpi kernel: DMA: 341*4kB (UMEC) 244*8kB (UMEC) 78*16kB (UMEC) 53*32kB (UEC) 7*64kB (UEC) 0*128kB 1*256kB (C) 0*512kB 0*1024kB 0*2048kB 121*4096kB (C) = 502580kB
Mar 19 23:16:10 rpi kernel: 88041 total pagecache pages
Mar 19 23:16:10 rpi kernel: 0 pages in swap cache
Mar 19 23:16:10 rpi kernel: Free swap = 0kB
Mar 19 23:16:10 rpi kernel: Total swap = 0kB
Mar 19 23:16:10 rpi kernel: 1012736 pages RAM
Mar 19 23:16:10 rpi kernel: 816128 pages HighMem/MovableOnly
Mar 19 23:16:10 rpi kernel: 30647 pages reserved
Mar 19 23:16:10 rpi kernel: 131072 pages cma reserved
Mar 19 23:16:10 rpi kernel: Tasks state (memory values in pages):
Mar 19 23:16:10 rpi kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Mar 19 23:16:10 rpi kernel: [ 211] 0 211 11787 3508 90112 0 -250 systemd-journal
Mar 19 23:16:10 rpi kernel: [ 212] 0 212 7071 1845 61440 0 -1000 systemd-udevd
Mar 19 23:16:10 rpi kernel: [ 217] 980 217 3846 1528 57344 0 0 systemd-network
Mar 19 23:16:10 rpi kernel: [ 315] 978 315 4570 2709 65536 0 0 systemd-resolve
Mar 19 23:16:10 rpi kernel: [ 320] 81 320 2835 1228 49152 0 -900 dbus-daemon
Mar 19 23:16:10 rpi kernel: [ 322] 0 322 5373 1459 53248 0 0 fake-hwclock-lo
Mar 19 23:16:10 rpi kernel: [ 325] 975 325 667 467 28672 0 0 dhcpcd
Mar 19 23:16:10 rpi kernel: [ 326] 0 326 628 388 28672 0 0 dhcpcd
Mar 19 23:16:10 rpi kernel: [ 327] 975 327 624 56 28672 0 0 dhcpcd
Mar 19 23:16:10 rpi kernel: [ 328] 975 328 624 56 28672 0 0 dhcpcd
Mar 19 23:16:10 rpi kernel: [ 330] 0 330 3670 1624 57344 0 0 systemd-logind
Mar 19 23:16:10 rpi kernel: [ 341] 967 341 5426 992 57344 0 0 chronyd
Mar 19 23:16:10 rpi kernel: [ 346] 0 346 1723 193 36864 0 0 sleep
Mar 19 23:16:10 rpi kernel: [ 354] 975 354 628 59 28672 0 0 dhcpcd
Mar 19 23:16:10 rpi kernel: [ 356] 975 356 628 59 28672 0 0 dhcpcd
Mar 19 23:16:10 rpi kernel: [ 405] 975 405 628 59 28672 0 0 dhcpcd
Mar 19 23:16:10 rpi kernel: [ 411] 134 411 128725 19759 421888 0 0 netdata
Mar 19 23:16:10 rpi kernel: [ 413] 0 413 23191 4982 163840 0 0 smbd
Mar 19 23:16:10 rpi kernel: [ 414] 0 414 2423 1525 49152 0 -1000 sshd
Mar 19 23:16:10 rpi kernel: [ 418] 0 418 1097 198 32768 0 0 agetty
Mar 19 23:16:10 rpi kernel: [ 419] 1000 419 4171 2334 57344 0 100 systemd
Mar 19 23:16:10 rpi kernel: [ 420] 1000 420 10235 940 81920 0 100 (sd-pam)
Mar 19 23:16:10 rpi kernel: [ 426] 134 426 15147 3055 73728 0 0 netdata
Mar 19 23:16:10 rpi kernel: [ 453] 975 453 628 59 28672 0 0 dhcpcd
Mar 19 23:16:10 rpi kernel: [ 540] 1000 540 172681 4227 81920 0 200 syncthing
Mar 19 23:16:10 rpi kernel: [ 674] 0 674 22563 2773 147456 0 0 smbd-notifyd
Mar 19 23:16:10 rpi kernel: [ 675] 0 675 22561 1294 143360 0 0 smbd-cleanupd
Mar 19 23:16:10 rpi kernel: [ 692] 1000 692 173209 8277 131072 0 200 syncthing
Mar 19 23:16:10 rpi kernel: [ 736] 134 736 4762 1182 45056 0 0 apps.plugin
Mar 19 23:16:10 rpi kernel: [ 741] 134 741 1023 731 28672 0 0 bash
Mar 19 23:16:10 rpi kernel: [ 770] 134 770 1851 449 40960 0 0 nfacct.plugin
Mar 19 23:16:10 rpi kernel: [ 775] 134 775 9862 6435 94208 0 0 python3
Mar 19 23:16:10 rpi kernel: [ 955] 975 955 628 59 28672 0 0 dhcpcd
Mar 19 23:16:10 rpi kernel: [ 1109] 1000 1109 105616 74262 720896 0 200 qbittorrent-nox
Mar 19 23:16:10 rpi kernel: [ 1141] 966 1141 5716 2761 73728 0 0 openvpn
Mar 19 23:16:10 rpi kernel: [ 1324] 62702 1324 528 172 32768 0 0 microsocks-sile
Mar 19 23:16:10 rpi kernel: [ 1403] 1000 1403 9605 5509 102400 0 0 weechat-headles
Mar 19 23:16:10 rpi kernel: [ 1928] 62514 1928 45266 10363 311296 0 0 node
Mar 19 23:16:10 rpi kernel: [ 2555] 65534 2555 23563 4378 163840 0 0 smbd[192.168.1.
Mar 19 23:16:10 rpi kernel: [ 2558] 0 2558 20791 3113 147456 0 0 samba-dcerpcd
Mar 19 23:16:10 rpi kernel: [ 2568] 0 2568 23731 5182 167936 0 0 rpcd_classic
Mar 19 23:16:10 rpi kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=qbittorrent-nox,pid=1109,uid=1000
Mar 19 23:16:10 rpi kernel: Out of memory: Killed process 1109 (qbittorrent-nox) total-vm:422464kB, anon-rss:271560kB, file-rss:25488kB, shmem-rss:0kB, UID:1000 pgtables:704kB oom_score_adj:200
Mar 19 23:16:10 rpi systemd[419]: qbt.service: Main process exited, code=killed, status=9/KILL
Mar 19 23:16:10 rpi systemd[419]: qbt.service: Failed with result 'signal'.
Mar 19 23:16:10 rpi systemd[419]: qbt.service: Consumed 1min 19.909s CPU time.
')

I'm running on the latest kernel, and I also checked all of the tmpfs mounts to see if any of them are using large amounts of data, they aren't.

Does anybody know what's happening? :cry:

This setup used to work perfectly a month ago and I did not make any significant changes to the configuration.
TheDcoder
 
Posts: 6
Joined: Wed Mar 01, 2023 7:38 pm

Re: [linux-rpi] OOM killer issues (armv7h)

Postby graysky » Sun Mar 19, 2023 6:52 pm

MOD NOTE - merged (again)
graysky
Developer
 
Posts: 1727
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000

Re: [linux-rpi] OOM killer issues (armv7h)

Postby TheDcoder » Sun Mar 19, 2023 6:58 pm

Someone requested a snippet of OOM's kill session before I disabled all memory overcommits today:

$this->bbcode_second_pass_code('', '
Mar 19 21:07:33 rpi kernel: apps.plugin invoked oom-killer: gfp_mask=0x40cd0(GFP_KERNEL|__GFP_COMP|__GFP_RECLAIMABLE), order=0, oom_score_adj=0
Mar 19 21:07:33 rpi kernel: CPU: 3 PID: 742 Comm: apps.plugin Tainted: G C 6.1.16-3-rpi-ARCH #1
Mar 19 21:07:33 rpi kernel: Hardware name: BCM2711
Mar 19 21:07:33 rpi kernel: unwind_backtrace from show_stack+0x18/0x1c
Mar 19 21:07:33 rpi kernel: show_stack from dump_stack_lvl+0x90/0xac
Mar 19 21:07:33 rpi kernel: dump_stack_lvl from dump_header+0x54/0x1fc
Mar 19 21:07:33 rpi kernel: dump_header from oom_kill_process+0x23c/0x248
Mar 19 21:07:33 rpi kernel: oom_kill_process from out_of_memory+0x218/0x34c
Mar 19 21:07:33 rpi kernel: out_of_memory from __alloc_pages+0xa98/0x1044
Mar 19 21:07:33 rpi kernel: __alloc_pages from new_slab+0x384/0x43c
Mar 19 21:07:33 rpi kernel: new_slab from ___slab_alloc+0x3e8/0xa0c
Mar 19 21:07:33 rpi kernel: ___slab_alloc from kmem_cache_alloc_lru+0x4fc/0x640
Mar 19 21:07:33 rpi kernel: kmem_cache_alloc_lru from proc_alloc_inode+0x24/0x5c
Mar 19 21:07:33 rpi kernel: proc_alloc_inode from alloc_inode+0x24/0xc8
Mar 19 21:07:33 rpi kernel: alloc_inode from new_inode+0x14/0x98
Mar 19 21:07:33 rpi kernel: new_inode from proc_pid_make_inode+0x2c/0xe8
Mar 19 21:07:33 rpi kernel: proc_pid_make_inode from proc_pid_make_base_inode.constprop.0+0x14/0x6c
Mar 19 21:07:33 rpi kernel: proc_pid_make_base_inode.constprop.0 from proc_pid_instantiate+0x1c/0xa4
Mar 19 21:07:33 rpi kernel: proc_pid_instantiate from proc_pid_lookup+0x84/0x128
Mar 19 21:07:33 rpi kernel: proc_pid_lookup from proc_root_lookup+0x24/0x40
Mar 19 21:07:33 rpi kernel: proc_root_lookup from __lookup_slow+0x90/0x138
Mar 19 21:07:33 rpi kernel: __lookup_slow from walk_component+0xf4/0x164
Mar 19 21:07:33 rpi kernel: walk_component from link_path_walk.part.0.constprop.0+0x250/0x3a8
Mar 19 21:07:33 rpi kernel: link_path_walk.part.0.constprop.0 from path_openat+0x94/0x111c
Mar 19 21:07:33 rpi kernel: path_openat from do_filp_open+0x98/0x134
Mar 19 21:07:33 rpi kernel: do_filp_open from do_sys_openat2+0xbc/0x184
Mar 19 21:07:33 rpi kernel: do_sys_openat2 from sys_openat+0x98/0xd4
Mar 19 21:07:33 rpi kernel: sys_openat from ret_fast_syscall+0x0/0x1c
Mar 19 21:07:33 rpi kernel: Exception stack(0xf0c79fa8 to 0xf0c79ff0)
Mar 19 21:07:33 rpi kernel: 9fa0: 00008000 01b2ef18 ffffff9c 01b2ef18 00008000 00000000
Mar 19 21:07:33 rpi kernel: 9fc0: 00008000 01b2ef18 b6fcece0 00000142 004c7788 b6a8bc2c 00000000 00038471
Mar 19 21:07:33 rpi kernel: 9fe0: 00000001 be96cf18 00000000 b69e7fec
Mar 19 21:07:33 rpi kernel: Mem-Info:
Mar 19 21:07:33 rpi kernel: active_anon:102012 inactive_anon:240 isolated_anon:0
active_file:38067 inactive_file:43797 isolated_file:0
unevictable:0 dirty:1122 writeback:0
slab_reclaimable:3923 slab_unreclaimable:7136
mapped:30550 shmem:260 pagetables:1001
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:773878 free_pcp:0 free_cma:125351
Mar 19 21:07:33 rpi kernel: Node 0 active_anon:408048kB inactive_anon:960kB active_file:152268kB inactive_file:175188kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:122200kB dirty:4488kB writeback:0kB shmem:1040kB writeback_tmp:0kB kernel_stack:1976kB pagetables:4004kB sec_pagetables:0kB all_unreclaimable? no
Mar 19 21:07:33 rpi kernel: DMA free:510656kB boost:0kB min:3232kB low:4040kB high:4848kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:556kB inactive_file:79128kB unevictable:0kB writepending:3688kB present:786432kB managed:663844kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:501404kB
Mar 19 21:07:33 rpi kernel: lowmem_reserve[]: 0 0 3188 3188
Mar 19 21:07:33 rpi kernel: DMA: 1371*4kB (UEC) 423*8kB (UEC) 37*16kB (UEC) 19*32kB (UEC) 6*64kB (C) 5*128kB (C) 3*256kB (C) 3*512kB (C) 2*1024kB (C) 0*2048kB 121*4096kB (C) = 511060kB
Mar 19 21:07:33 rpi kernel: 82134 total pagecache pages
Mar 19 21:07:33 rpi kernel: 0 pages in swap cache
Mar 19 21:07:33 rpi kernel: Free swap = 0kB
Mar 19 21:07:33 rpi kernel: Total swap = 0kB
Mar 19 21:07:33 rpi kernel: 1012736 pages RAM
Mar 19 21:07:33 rpi kernel: 816128 pages HighMem/MovableOnly
Mar 19 21:07:33 rpi kernel: 30647 pages reserved
Mar 19 21:07:33 rpi kernel: 131072 pages cma reserved
Mar 19 21:07:33 rpi kernel: Tasks state (memory values in pages):
Mar 19 21:07:33 rpi kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Mar 19 21:07:33 rpi kernel: [ 212] 0 212 11754 3504 90112 0 -250 systemd-journal
Mar 19 21:07:33 rpi kernel: [ 213] 0 213 7070 1838 57344 0 -1000 systemd-udevd
Mar 19 21:07:33 rpi kernel: [ 216] 980 216 3846 1527 53248 0 0 systemd-network
Mar 19 21:07:33 rpi kernel: [ 315] 978 315 4538 2806 61440 0 0 systemd-resolve
Mar 19 21:07:33 rpi kernel: [ 319] 81 319 2836 1230 45056 0 -900 dbus-daemon
Mar 19 21:07:33 rpi kernel: [ 321] 0 321 5373 1459 61440 0 0 fake-hwclock-lo
Mar 19 21:07:33 rpi kernel: [ 324] 975 324 667 467 28672 0 0 dhcpcd
Mar 19 21:07:33 rpi kernel: [ 325] 0 325 628 388 32768 0 0 dhcpcd
Mar 19 21:07:33 rpi kernel: [ 326] 975 326 624 56 28672 0 0 dhcpcd
Mar 19 21:07:33 rpi kernel: [ 327] 975 327 624 56 28672 0 0 dhcpcd
Mar 19 21:07:33 rpi kernel: [ 328] 0 328 3670 1623 57344 0 0 systemd-logind
Mar 19 21:07:33 rpi kernel: [ 343] 967 343 5426 997 57344 0 0 chronyd
Mar 19 21:07:33 rpi kernel: [ 353] 975 353 628 59 32768 0 0 dhcpcd
Mar 19 21:07:33 rpi kernel: [ 355] 975 355 628 59 32768 0 0 dhcpcd
Mar 19 21:07:33 rpi kernel: [ 404] 975 404 628 59 32768 0 0 dhcpcd
Mar 19 21:07:33 rpi kernel: [ 406] 62514 406 45297 10164 315392 0 0 node
Mar 19 21:07:33 rpi kernel: [ 410] 134 410 129539 20640 425984 0 0 netdata
Mar 19 21:07:33 rpi kernel: [ 412] 0 412 22874 4694 163840 0 0 smbd
Mar 19 21:07:33 rpi kernel: [ 413] 0 413 2422 1601 45056 0 -1000 sshd
Mar 19 21:07:33 rpi kernel: [ 417] 0 417 1097 199 32768 0 0 agetty
Mar 19 21:07:33 rpi kernel: [ 418] 1000 418 4175 2327 61440 0 100 systemd
Mar 19 21:07:33 rpi kernel: [ 419] 1000 419 10232 971 77824 0 100 (sd-pam)
Mar 19 21:07:33 rpi kernel: [ 425] 134 425 15147 3055 77824 0 0 netdata
Mar 19 21:07:33 rpi kernel: [ 479] 1000 479 172681 4122 81920 0 200 syncthing
Mar 19 21:07:33 rpi kernel: [ 625] 1000 625 173145 7804 131072 0 200 syncthing
Mar 19 21:07:33 rpi kernel: [ 631] 0 631 22313 2763 151552 0 0 smbd-notifyd
Mar 19 21:07:33 rpi kernel: [ 632] 0 632 22311 1306 143360 0 0 cleanupd
Mar 19 21:07:33 rpi kernel: [ 704] 975 704 628 59 32768 0 0 dhcpcd
Mar 19 21:07:33 rpi kernel: [ 742] 134 742 4718 1182 53248 0 0 apps.plugin
Mar 19 21:07:33 rpi kernel: [ 755] 134 755 1851 449 45056 0 0 nfacct.plugin
Mar 19 21:07:33 rpi kernel: [ 760] 134 760 9874 6452 98304 0 0 python3
Mar 19 21:07:33 rpi kernel: [ 961] 975 961 628 59 32768 0 0 dhcpcd
Mar 19 21:07:33 rpi kernel: [ 1098] 966 1098 5714 2766 73728 0 0 openvpn
Mar 19 21:07:33 rpi kernel: [ 2612] 1000 2612 105893 74383 733184 0 200 qbittorrent-nox
Mar 19 21:07:33 rpi kernel: [ 4681] 0 4681 27332 4221 176128 0 0 smbd
Mar 19 21:07:33 rpi kernel: [ 4685] 0 4685 20753 3097 131072 0 0 samba-dcerpcd
Mar 19 21:07:33 rpi kernel: [ 4695] 0 4695 23680 5188 163840 0 0 rpcd_classic
Mar 19 21:07:33 rpi kernel: [ 5390] 0 5390 1723 194 40960 0 0 sleep
Mar 19 21:07:33 rpi kernel: [ 5402] 134 5402 1023 732 28672 0 0 bash
Mar 19 21:07:33 rpi kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=qbittorrent-nox,pid=2612,uid=1000
Mar 19 21:07:33 rpi kernel: Out of memory: Killed process 2612 (qbittorrent-nox) total-vm:423572kB, anon-rss:271584kB, file-rss:25948kB, shmem-rss:0kB, UID:1000 pgtables:716kB oom_score_adj:200
Mar 19 21:07:33 rpi systemd[418]: qbt.service: Main process exited, code=killed, status=9/KILL
Mar 19 21:07:33 rpi systemd[418]: qbt.service: Failed with result 'signal'.
Mar 19 21:07:33 rpi systemd[418]: qbt.service: Consumed 3min 3.716s CPU time.
')
TheDcoder
 
Posts: 6
Joined: Wed Mar 01, 2023 7:38 pm

Re: [linux-rpi] OOM killer issues (armv7h)

Postby graysky » Sun Mar 19, 2023 7:19 pm

$this->bbcode_second_pass_quote('graysky', 'C')an someone open an issue upstream.


Likely not a kernel config thing. Upstream may have insights.
graysky
Developer
 
Posts: 1727
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000

Re: [linux-rpi] OOM killer issues (armv7h)

Postby amhairghin » Mon Mar 20, 2023 2:47 pm

If you find it useful. In my case I have not touched any of the two files.

$this->bbcode_second_pass_code('', '
$ cat /boot/config.txt
# See /boot/overlays/README for all available options

dtoverlay=vc4-kms-v3d
initramfs initramfs-linux.img followkernel

# Automatically load overlays for detected DSI displays
display_auto_detect=1

# uncomment if hdmi display is not detected and composite is being output
#hdmi_force_hotplug=1

# Uncomment some or all of these to enable the optional hardware interfaces
#dtparam=i2c_arm=on
#dtparam=i2s=on
#dtparam=spi=on

# Uncomment this to enable infrared communication.
#dtoverlay=gpio-ir,gpio_pin=17
#dtoverlay=gpio-ir-tx,gpio_pin=18

# Automatically load overlays for detected cameras
camera_auto_detect=1

# Uncomment to enable bluetooth
#dtparam=krnbt=on

[cm4]
# Enable host mode on the 2711 built-in XHCI USB controller.
# This line should be removed if the legacy DWC2 controller is required
# (e.g. for USB device mode) or if USB support is not required.
otg_mode=1

[pi4]
# Run as fast as firmware / board allows
arm_boost=1
')

$this->bbcode_second_pass_code('', '
$ cat /boot/cmdline.txt
root=/dev/mmcblk0p2 rw rootwait console=serial0,115200 console=tty1 fsck.repair=yes
')
amhairghin
 
Posts: 8
Joined: Tue Sep 19, 2017 6:51 am

Re: [linux-rpi] OOM killer issues (armv7h)

Postby gavinbeatty » Wed Mar 22, 2023 2:56 pm

gavinbeatty
 
Posts: 9
Joined: Fri Mar 17, 2023 9:54 pm

Re: [linux-rpi] OOM killer issues (armv7h)

Postby gavinbeatty » Wed Mar 22, 2023 8:23 pm

Thanks to help from upstream, a partial workaround is to set /sys/kernel/mm/lru_gen/enabled to 0, and append ",cma-320" to dtoverlay.

So run:

$this->bbcode_second_pass_code('', 'systemd-tmpfiles --create sysfs-memory-fix.conf')
with the following config:
$this->bbcode_second_pass_code('', '
# /etc/tmpfiles.d/sysfs-memory-fix.conf
w- /sys/kernel/mm/lru_gen/enabled - - - - 0
')

And change /boot/config.txt from dtoverlay=vc4-kms-v3d to dtoverlay=vc4-kms-v3d,cma-320

With both of these changes, occasionally a syscall will still __alloc_pages and hit out_of_memory, but it is far less frequent. So far it has only happened for network syscalls, instead of filesystem ones, so it has been far less disruptive. Even pacman used to trigger it all the time while upgrading packages, but no more.

@graysky Their comments about the "MGLRU" might point to something that could be fixed in kernel build config, or something unique to archlinuxarm but not RPiOS, but that's just speculation.

These SLAB allocator configs seem to be what they're referring to.
$this->bbcode_second_pass_code('', '
CONFIG_LRU_GEN=y
CONFIG_LRU_GEN_ENABLED=y
# CONFIG_LRU_GEN_STATS is not set
')
gavinbeatty
 
Posts: 9
Joined: Fri Mar 17, 2023 9:54 pm

PreviousNext

Return to U-Boot/Kernel

Who is online

Users browsing this forum: No registered users and 5 guests