[linux-rpi] OOM killer issues (armv7h)

[zxcv]

Hi everyone.
There are some issues with linux-rpi-6.1.11-1 (and probably earlier packages), Out Of Memory Killer (OOM) is killing processes even though there is still enough free ram (1G used, 3G free).

I spent a week trying to tweak my systemd units memory limits and diagnose this issue until it dawned on me that OOM killer was invoked by the kernel. Downgraded to random 5.15 kernel (5.15.83-2) and system finally does not try to self destruct.

[graysky]

Thank you for the report. Which RPi model?

[zxcv]

Should have posted that:
Raspberry Pi 4 Model B Rev 1.2
4GB version.

[graysky]

I am not sure if there is a kernel config option that isn't enabled on the armv7h kernel driving this behavior or if it is something in the 6.1,y branch of kernels. One question that is unrelated to the behavior you're experiencing is why are you running armv7h on RPi4/4G? You would benefit from the 64-bit kernel/userland on this hardware.

[amhairghin]

I have exactly the same problem with exactly the same model. Also the oom-killer is extremely aggressive, it not only "kills" the process that consumes the most memory but it kills several processes at the same time.

This is something that I had never done before, so it fits me perfectly that the cause is the change of Kernel from 5.15 to 6.1.

Have any modifications been made to the configuration that might account for this?

[graysky]

Please post the output of uname -a

[amhairghin]

In my case:

$this->bbcode_second_pass_code('', 'Linux sigaba 6.1.15-2-rpi-ARCH #1 SMP Wed Mar 8 07:24:27 MST 2023 armv7l GNU/Linux')

And CPU info
$this->bbcode_second_pass_code('', '
processor : 0
model name : ARMv7 Processor rev 3 (v7l)
BogoMIPS : 270.00
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd08
CPU revision : 3

processor : 1
model name : ARMv7 Processor rev 3 (v7l)
BogoMIPS : 270.00
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd08
CPU revision : 3

processor : 2
model name : ARMv7 Processor rev 3 (v7l)
BogoMIPS : 270.00
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd08
CPU revision : 3

processor : 3
model name : ARMv7 Processor rev 3 (v7l)
BogoMIPS : 270.00
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd08
CPU revision : 3

Hardware : BCM2711
Revision : c03111
Serial : 10000000ab9b6234
Model : Raspberry Pi 4 Model B Rev 1.1
')

[graysky]

OK, just wanted to be sure this was the linux-rpi kernel and not the vanilla kernel. I looked into this but am not seeing a config option that I missed that might be driving this.

[robg]

It might be of benefit to see contents of /boot/config.txt and /boot/cmdline.txt. Have you perhaps tried Raspberry Pi OS (with the same contents of the aforementioned files) for comparison?

[jswcom]

My Raspberry Pi 4B is dying at least once a day, when it does something (for example, when backup job starts). The memory usage on this one is always below 15%, usually below 10%.

Below is the log, including an output from "free -hw", which logged 7 seconds before OOM. As you can see, there's a lot of RAM available. "net-restart.sh" is just a simple bash script, which restarts wi-fi when it loses connection. The most complicated thing it does is ping, so there's no reason for it to cause OOM when there's more than 3 GiB free. Sometimes it's triggered by PostgreSQL vacuum service, sometimes rsync-based backup. When it goes OOM, it just starts killing one process after another until it dies completely.

I have upgraded the kernel (and other stuff) few times since this started to happen (in December). And there was no SW change before it started. Weird is that it started to happen this week on my other Pi 4B as well. The configuration is similar, but it's doing completely different work.

Btw, I have also tried to add swap (2 GiB), but it didn't help.

$this->bbcode_second_pass_code('', '23:00:02 free[10890]: total used free shared buffers cache available
23:00:02 free[10890]: Mem: 3,7Gi 82Mi 3,2Gi 2,0Mi 0,0Ki 442Mi 3,6Gi
23:00:02 free[10890]: Swap: 0B 0B 0B

23:00:09 kernel: oom_kill_process: 13 callbacks suppressed
23:00:09 kernel: net-restart.sh invoked oom-killer: gfp_mask=0x400dc0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), order=0, oom_score_adj=0
23:00:09 kernel: CPU: 2 PID: 10992 Comm: net-restart.sh Tainted: G C 6.1.14-1-rpi-ARCH #1
23:00:09 kernel: Hardware name: BCM2711
23:00:09 kernel: unwind_backtrace from show_stack+0x18/0x1c
23:00:09 kernel: show_stack from dump_stack_lvl+0x90/0xac
23:00:09 kernel: dump_stack_lvl from dump_header+0x54/0x1fc
23:00:09 kernel: dump_header from oom_kill_process+0x23c/0x248
23:00:09 kernel: oom_kill_process from out_of_memory+0x218/0x34c
23:00:09 kernel: out_of_memory from __alloc_pages+0xa98/0x1044
23:00:09 kernel: __alloc_pages from __pmd_alloc+0x3c/0x1d8
23:00:09 kernel: __pmd_alloc from copy_page_range+0xcac/0xcc4
23:00:09 kernel: copy_page_range from dup_mm+0x440/0x5a4
23:00:09 kernel: dup_mm from copy_process+0xda0/0x164c
23:00:09 kernel: copy_process from kernel_clone+0xac/0x3a8
23:00:09 kernel: kernel_clone from sys_clone+0x78/0x9c
23:00:09 kernel: sys_clone from ret_fast_syscall+0x0/0x1c
23:00:09 kernel: Exception stack(0xf08b1fa8 to 0xf08b1ff0)
23:00:09 kernel: 1fa0: b6fd0088 00000001 01200011 00000000 00000000 00000000
23:00:09 kernel: 1fc0: b6fd0088 00000001 b6efae58 00000078 bea210fc 0055d2bc bea2107c 005844e0
23:00:09 kernel: 1fe0: b6fd05a0 bea20f08 b6e2d260 b6e2d684
23:00:09 kernel: Mem-Info:
23:00:09 kernel: active_anon:7451 inactive_anon:603 isolated_anon:0
active_file:39567 inactive_file:70065 isolated_file:0
unevictable:0 dirty:143 writeback:0
slab_reclaimable:3166 slab_unreclaimable:6791
mapped:23163 shmem:594 pagetables:267
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:848488 free_pcp:30 free_cma:80063
23:00:09 kernel: Node 0 active_anon:29804kB inactive_anon:2412kB active_file:158268kB inactive_file:280260kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:92652kB dirty:572kB writeback:0kB shmem:2376kB writeback_tmp:0kB kernel_stack:2360kB pagetables:1068kB sec_pagetab>
23:00:09 kernel: DMA free:323468kB boost:0kB min:3236kB low:4044kB high:4852kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:8076kB inactive_file:279068kB unevictable:0kB writepending:0kB present:786432kB managed:664228kB mlocked:0kB bounce:0kB free_pcp:120kB >
23:00:09 kernel: lowmem_reserve[]: 0 0 3188 3188
23:00:09 kernel: DMA: 143*4kB (UMEC) 119*8kB (UMEC) 68*16kB (UMEC) 23*32kB (UEC) 1*64kB (C) 1*128kB (C) 0*256kB 1*512kB (C) 0*1024kB 0*2048kB 78*4096kB (C) = 323540kB
23:00:09 kernel: 110236 total pagecache pages
23:00:09 kernel: 0 pages in swap cache
23:00:09 kernel: Free swap = 0kB
23:00:09 kernel: Total swap = 0kB
23:00:09 kernel: 1012736 pages RAM
23:00:09 kernel: 816128 pages HighMem/MovableOnly
23:00:09 kernel: 30551 pages reserved
23:00:09 kernel: 81920 pages cma reserved
23:00:09 kernel: Tasks state (memory values in pages):
23:00:09 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
23:00:09 kernel: [ 242] 0 242 12050 4296 98304 0 -250 systemd-journal
23:00:09 kernel: [ 243] 0 243 7022 1837 61440 0 -1000 systemd-udevd
23:00:09 kernel: [ 516] 81 516 2843 1047 49152 0 -900 dbus-daemon
23:00:09 kernel: [ 550] 0 550 2422 1664 45056 0 -1000 sshd
23:00:09 kernel: [ 554] 0 554 196576 7435 167936 0 -999 containerd
23:00:09 kernel: [ 651] 0 651 203978 13307 245760 0 -500 dockerd
23:00:09 kernel: [ 10882] 978 10882 4543 2764 65536 0 0 systemd-resolve
23:00:09 kernel: [ 10888] 0 10888 1097 201 36864 0 0 agetty
23:00:09 kernel: [ 10889] 977 10889 6022 965 65536 0 0 systemd-timesyn
23:00:09 kernel: [ 10890] 0 10890 2676 341 49152 0 0 free
23:00:09 kernel: [ 10897] 0 10897 3543 1493 57344 0 0 systemd-logind
23:00:09 kernel: [ 10992] 0 10992 2169 824 40960 0 0 net-restart.sh
23:00:09 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=net-restart.service,mems_allowed=0,global_oom,task_memcg=/,task=systemd-resolve,pid=10882,uid=978
23:00:09 kernel: Out of memory: Killed process 10882 (systemd-resolve) total-vm:18172kB, anon-rss:1548kB, file-rss:9508kB, shmem-rss:0kB, UID:978 pgtables:64kB oom_score_adj:0
')