Hi,
The OOM Killer has gone rouge in my system since a couple of weeks, it keeps killing processes at random even though there is plenty of free RAM (> 85% always).
It has especially taken a liking to killing qBittorrent (nox). If I keep the system running for long enough, it will even kill systemd and make the system useless essentially.
Here's a typical kill session:
$this->bbcode_second_pass_code('', '
Mar 19 23:16:10 rpi kernel: bash invoked oom-killer: gfp_mask=0x400dc0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), order=0, oom_score_adj=0
Mar 19 23:16:10 rpi kernel: CPU: 3 PID: 741 Comm: bash Tainted: G C 6.1.19-2-rpi-ARCH #1
Mar 19 23:16:10 rpi kernel: Hardware name: BCM2711
Mar 19 23:16:10 rpi kernel: unwind_backtrace from show_stack+0x18/0x1c
Mar 19 23:16:10 rpi kernel: show_stack from dump_stack_lvl+0x90/0xac
Mar 19 23:16:10 rpi kernel: dump_stack_lvl from dump_header+0x54/0x1fc
Mar 19 23:16:10 rpi kernel: dump_header from oom_kill_process+0x23c/0x248
Mar 19 23:16:10 rpi kernel: oom_kill_process from out_of_memory+0x218/0x34c
Mar 19 23:16:10 rpi kernel: out_of_memory from __alloc_pages+0xa98/0x1044
Mar 19 23:16:10 rpi kernel: __alloc_pages from __pmd_alloc+0x3c/0x1d8
Mar 19 23:16:10 rpi kernel: __pmd_alloc from pgd_alloc+0x2e0/0x32c
Mar 19 23:16:10 rpi kernel: pgd_alloc from mm_init+0x100/0x1cc
Mar 19 23:16:10 rpi kernel: mm_init from dup_mm+0x60/0x5a4
Mar 19 23:16:10 rpi kernel: dup_mm from copy_process+0xda0/0x164c
Mar 19 23:16:10 rpi kernel: copy_process from kernel_clone+0xac/0x3a8
Mar 19 23:16:10 rpi kernel: kernel_clone from sys_clone+0x78/0x9c
Mar 19 23:16:10 rpi kernel: sys_clone from ret_fast_syscall+0x0/0x1c
Mar 19 23:16:10 rpi kernel: Exception stack(0xf0c31fa8 to 0xf0c31ff0)
Mar 19 23:16:10 rpi kernel: 1fa0: b6fa8088 00000001 01200011 00000000 00000000 00000000
Mar 19 23:16:10 rpi kernel: 1fc0: b6fa8088 00000001 b6ebae58 00000078 befc624c 0057d2bc befc61cc 005a44e0
Mar 19 23:16:10 rpi kernel: 1fe0: b6fa85a0 befc6058 b6ded260 b6ded684
Mar 19 23:16:10 rpi kernel: Mem-Info:
Mar 19 23:16:10 rpi kernel: active_anon:104352 inactive_anon:245 isolated_anon:0
active_file:39467 inactive_file:48297 isolated_file:0
unevictable:0 dirty:3095 writeback:0
slab_reclaimable:3924 slab_unreclaimable:7175
mapped:31985 shmem:265 pagetables:1022
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:764189 free_pcp:663 free_cma:124679
Mar 19 23:16:10 rpi kernel: Node 0 active_anon:417408kB inactive_anon:980kB active_file:157868kB inactive_file:193188kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:127940kB dirty:12380kB writeback:0kB shmem:1060kB writeback_tmp:0kB kernel_stack:1984kB pagetables:4088kB sec_pagetables:0kB all_unreclaimable? yes
Mar 19 23:16:10 rpi kernel: DMA free:502580kB boost:7272kB min:10504kB low:11312kB high:12120kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:712kB inactive_file:84296kB unevictable:0kB writepending:11944kB present:786432kB managed:663844kB mlocked:0kB bounce:0kB free_pcp:2652kB local_pcp:520kB free_cma:498716kB
Mar 19 23:16:10 rpi kernel: lowmem_reserve[]: 0 0 3188 3188
Mar 19 23:16:10 rpi kernel: DMA: 341*4kB (UMEC) 244*8kB (UMEC) 78*16kB (UMEC) 53*32kB (UEC) 7*64kB (UEC) 0*128kB 1*256kB (C) 0*512kB 0*1024kB 0*2048kB 121*4096kB (C) = 502580kB
Mar 19 23:16:10 rpi kernel: 88041 total pagecache pages
Mar 19 23:16:10 rpi kernel: 0 pages in swap cache
Mar 19 23:16:10 rpi kernel: Free swap = 0kB
Mar 19 23:16:10 rpi kernel: Total swap = 0kB
Mar 19 23:16:10 rpi kernel: 1012736 pages RAM
Mar 19 23:16:10 rpi kernel: 816128 pages HighMem/MovableOnly
Mar 19 23:16:10 rpi kernel: 30647 pages reserved
Mar 19 23:16:10 rpi kernel: 131072 pages cma reserved
Mar 19 23:16:10 rpi kernel: Tasks state (memory values in pages):
Mar 19 23:16:10 rpi kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Mar 19 23:16:10 rpi kernel: [ 211] 0 211 11787 3508 90112 0 -250 systemd-journal
Mar 19 23:16:10 rpi kernel: [ 212] 0 212 7071 1845 61440 0 -1000 systemd-udevd
Mar 19 23:16:10 rpi kernel: [ 217] 980 217 3846 1528 57344 0 0 systemd-network
Mar 19 23:16:10 rpi kernel: [ 315] 978 315 4570 2709 65536 0 0 systemd-resolve
Mar 19 23:16:10 rpi kernel: [ 320] 81 320 2835 1228 49152 0 -900 dbus-daemon
Mar 19 23:16:10 rpi kernel: [ 322] 0 322 5373 1459 53248 0 0 fake-hwclock-lo
Mar 19 23:16:10 rpi kernel: [ 325] 975 325 667 467 28672 0 0 dhcpcd
Mar 19 23:16:10 rpi kernel: [ 326] 0 326 628 388 28672 0 0 dhcpcd
Mar 19 23:16:10 rpi kernel: [ 327] 975 327 624 56 28672 0 0 dhcpcd
Mar 19 23:16:10 rpi kernel: [ 328] 975 328 624 56 28672 0 0 dhcpcd
Mar 19 23:16:10 rpi kernel: [ 330] 0 330 3670 1624 57344 0 0 systemd-logind
Mar 19 23:16:10 rpi kernel: [ 341] 967 341 5426 992 57344 0 0 chronyd
Mar 19 23:16:10 rpi kernel: [ 346] 0 346 1723 193 36864 0 0 sleep
Mar 19 23:16:10 rpi kernel: [ 354] 975 354 628 59 28672 0 0 dhcpcd
Mar 19 23:16:10 rpi kernel: [ 356] 975 356 628 59 28672 0 0 dhcpcd
Mar 19 23:16:10 rpi kernel: [ 405] 975 405 628 59 28672 0 0 dhcpcd
Mar 19 23:16:10 rpi kernel: [ 411] 134 411 128725 19759 421888 0 0 netdata
Mar 19 23:16:10 rpi kernel: [ 413] 0 413 23191 4982 163840 0 0 smbd
Mar 19 23:16:10 rpi kernel: [ 414] 0 414 2423 1525 49152 0 -1000 sshd
Mar 19 23:16:10 rpi kernel: [ 418] 0 418 1097 198 32768 0 0 agetty
Mar 19 23:16:10 rpi kernel: [ 419] 1000 419 4171 2334 57344 0 100 systemd
Mar 19 23:16:10 rpi kernel: [ 420] 1000 420 10235 940 81920 0 100 (sd-pam)
Mar 19 23:16:10 rpi kernel: [ 426] 134 426 15147 3055 73728 0 0 netdata
Mar 19 23:16:10 rpi kernel: [ 453] 975 453 628 59 28672 0 0 dhcpcd
Mar 19 23:16:10 rpi kernel: [ 540] 1000 540 172681 4227 81920 0 200 syncthing
Mar 19 23:16:10 rpi kernel: [ 674] 0 674 22563 2773 147456 0 0 smbd-notifyd
Mar 19 23:16:10 rpi kernel: [ 675] 0 675 22561 1294 143360 0 0 smbd-cleanupd
Mar 19 23:16:10 rpi kernel: [ 692] 1000 692 173209 8277 131072 0 200 syncthing
Mar 19 23:16:10 rpi kernel: [ 736] 134 736 4762 1182 45056 0 0 apps.plugin
Mar 19 23:16:10 rpi kernel: [ 741] 134 741 1023 731 28672 0 0 bash
Mar 19 23:16:10 rpi kernel: [ 770] 134 770 1851 449 40960 0 0 nfacct.plugin
Mar 19 23:16:10 rpi kernel: [ 775] 134 775 9862 6435 94208 0 0 python3
Mar 19 23:16:10 rpi kernel: [ 955] 975 955 628 59 28672 0 0 dhcpcd
Mar 19 23:16:10 rpi kernel: [ 1109] 1000 1109 105616 74262 720896 0 200 qbittorrent-nox
Mar 19 23:16:10 rpi kernel: [ 1141] 966 1141 5716 2761 73728 0 0 openvpn
Mar 19 23:16:10 rpi kernel: [ 1324] 62702 1324 528 172 32768 0 0 microsocks-sile
Mar 19 23:16:10 rpi kernel: [ 1403] 1000 1403 9605 5509 102400 0 0 weechat-headles
Mar 19 23:16:10 rpi kernel: [ 1928] 62514 1928 45266 10363 311296 0 0 node
Mar 19 23:16:10 rpi kernel: [ 2555] 65534 2555 23563 4378 163840 0 0 smbd[192.168.1.
Mar 19 23:16:10 rpi kernel: [ 2558] 0 2558 20791 3113 147456 0 0 samba-dcerpcd
Mar 19 23:16:10 rpi kernel: [ 2568] 0 2568 23731 5182 167936 0 0 rpcd_classic
Mar 19 23:16:10 rpi kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=qbittorrent-nox,pid=1109,uid=1000
Mar 19 23:16:10 rpi kernel: Out of memory: Killed process 1109 (qbittorrent-nox) total-vm:422464kB, anon-rss:271560kB, file-rss:25488kB, shmem-rss:0kB, UID:1000 pgtables:704kB oom_score_adj:200
Mar 19 23:16:10 rpi systemd[419]: qbt.service: Main process exited, code=killed, status=9/KILL
Mar 19 23:16:10 rpi systemd[419]: qbt.service: Failed with result 'signal'.
Mar 19 23:16:10 rpi systemd[419]: qbt.service: Consumed 1min 19.909s CPU time.
')
I'm running on the latest kernel, and I also checked all of the tmpfs mounts to see if any of them are using large amounts of data, they aren't.
Does anybody know what's happening?
This setup used to work perfectly a month ago and I did not make any significant changes to the configuration.