Random OOM errors and Invalid argument w/ systmctl disable

Ask questions about Arch Linux ARM. Please search before making a new topic.

Random OOM errors and Invalid argument w/ systmctl disable

Postby uhhyeahbret » Thu Mar 16, 2023 3:18 am

After updating today (raspi4 arm7), I started getting OOM killed errors on some processes, despite not really using much memory at all.

The process that was triggering the oom killer was prometheus-node-exporter.service, which I was using to collect stats on my hosts. Not critical, so I went to disable the service and I got the following error:

$this->bbcode_second_pass_code('', 'sudo systemctl disable prometheus-node-exporter.service
Failed to disable unit: Invalid argument')

I tried disabling some other services and got the same error as well.

I tried to disable a service on another host that I also recently updated and am getting the same error.

It seems like a recent update has broken something about systems ability to disable services.

Anyone else seeing this?
Last edited by uhhyeahbret on Sun Mar 19, 2023 7:19 pm, edited 1 time in total.
uhhyeahbret
 
Posts: 12
Joined: Thu Sep 03, 2015 11:10 am

Re: Failed to disable unit: Invalid argument w/ systmctl dis

Postby sgk60 » Thu Mar 16, 2023 11:27 am

sgk60
 
Posts: 2
Joined: Fri Feb 11, 2022 10:18 am

Re: Failed to disable unit: Invalid argument w/ systmctl dis

Postby uhhyeahbret » Thu Mar 16, 2023 3:17 pm

Thanks, seemed like a bug. I wonder if the oom killing is related.
uhhyeahbret
 
Posts: 12
Joined: Thu Sep 03, 2015 11:10 am

Re: Failed to disable unit: Invalid argument w/ systmctl dis

Postby uhhyeahbret » Thu Mar 16, 2023 3:21 pm

This was the oom killer message I would receive. Seemed to affect a variety of different processes:

Whats odd is my swap file appears totally un-utilized.

$this->bbcode_second_pass_code('', '
ar 16 15:16:46 ace systemd[1]: user@1001.service: Failed with result 'signal'.
Mar 16 15:16:46 ace systemd[1]: user@1001.service: Main process exited, code=killed, status=9/KILL
Mar 16 15:16:46 ace (sd-pam)[1427]: pam_warn(systemd-user:setcred): function=[pam_sm_setcred] flags=0x8004 service=[systemd-user] terminal=[] user=[bret] ruser=[<unknown>] rhost=[<unknown>]
Mar 16 15:16:46 ace kernel: Out of memory: Killed process 1426 (systemd) total-vm:16592kB, anon-rss:1132kB, file-rss:8116kB, shmem-rss:0kB, UID:1001 pgtables:56kB oom_score_adj:100
Mar 16 15:16:46 ace kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=systemd,pid=1426,uid=1001
Mar 16 15:16:46 ace kernel: [ 1469] 1001 1469 227446 9429 184320 0 0 yay
Mar 16 15:16:46 ace kernel: [ 1455] 1001 1455 2198 969 40960 0 0 bash
Mar 16 15:16:46 ace kernel: [ 1453] 1001 1453 3216 889 40960 0 0 tmux: server
Mar 16 15:16:46 ace kernel: [ 1451] 1001 1451 3147 1000 45056 0 0 tmux: client
Mar 16 15:16:46 ace kernel: [ 1435] 1001 1435 2194 1002 40960 0 0 bash
Mar 16 15:16:46 ace kernel: [ 1434] 1001 1434 3549 1588 53248 0 0 sshd
Mar 16 15:16:46 ace kernel: [ 1427] 1001 1427 10577 1226 77824 0 100 (sd-pam)
Mar 16 15:16:46 ace kernel: [ 1426] 1001 1426 4148 2312 57344 0 100 systemd
Mar 16 15:16:46 ace kernel: [ 1422] 0 1422 3509 2245 57344 0 0 sshd
Mar 16 15:16:46 ace kernel: [ 984] 971 984 10550 2542 94208 0 0 Plex Tuner Serv
Mar 16 15:16:46 ace kernel: [ 942] 971 942 11611 6795 98304 0 0 Plex Script Hos
Mar 16 15:16:46 ace kernel: [ 919] 971 919 33030 11109 315392 0 0 Plex Media Serv
Mar 16 15:16:46 ace kernel: [ 433] 33 433 2665 918 49152 0 0 nginx
Mar 16 15:16:46 ace kernel: [ 432] 33 432 2665 918 49152 0 0 nginx
Mar 16 15:16:46 ace kernel: [ 431] 33 431 2665 918 49152 0 0 nginx
Mar 16 15:16:46 ace kernel: [ 430] 33 430 2665 918 49152 0 0 nginx
Mar 16 15:16:46 ace kernel: [ 429] 0 429 2291 189 40960 0 0 nginx
Mar 16 15:16:46 ace kernel: [ 353] 0 353 1097 199 36864 0 0 agetty
Mar 16 15:16:46 ace kernel: [ 350] 0 350 2422 1624 45056 0 -1000 sshd
Mar 16 15:16:46 ace kernel: [ 347] 975 347 2942 130 40960 0 0 avahi-daemon
Mar 16 15:16:46 ace kernel: [ 346] 981 346 3874 1788 53248 0 0 systemd-network
Mar 16 15:16:46 ace kernel: [ 343] 0 343 3719 1635 57344 0 0 systemd-logind
Mar 16 15:16:46 ace kernel: [ 341] 81 341 2949 1252 49152 0 -900 dbus-daemon
Mar 16 15:16:46 ace kernel: [ 340] 975 340 3052 1163 53248 0 0 avahi-daemon
Mar 16 15:16:46 ace kernel: [ 319] 0 319 3351 426 32768 0 -1000 auditd
Mar 16 15:16:46 ace kernel: [ 317] 979 317 6075 1625 61440 0 0 systemd-timesyn
Mar 16 15:16:46 ace kernel: [ 316] 980 316 5147 2789 65536 0 0 systemd-resolve
Mar 16 15:16:46 ace kernel: [ 315] 0 315 12192 1058 73728 0 0 rngd
Mar 16 15:16:46 ace kernel: [ 216] 0 216 6984 1802 65536 0 -1000 systemd-udevd
Mar 16 15:16:46 ace kernel: [ 203] 0 203 15881 2065 86016 0 -250 systemd-journal
Mar 16 15:16:46 ace kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Mar 16 15:16:46 ace kernel: Tasks state (memory values in pages):
Mar 16 15:16:46 ace kernel: 131072 pages cma reserved
Mar 16 15:16:46 ace kernel: 39864 pages reserved
Mar 16 15:16:46 ace kernel: 1864704 pages HighMem/MovableOnly
Mar 16 15:16:46 ace kernel: 2061312 pages RAM
Mar 16 15:16:46 ace kernel: Total swap = 6143996kB
Mar 16 15:16:46 ace kernel: Free swap = 6143996kB
Mar 16 15:16:46 ace kernel: 0 pages in swap cache
Mar 16 15:16:46 ace kernel: 70759 total pagecache pages
Mar 16 15:16:46 ace kernel: DMA: 3*4kB (C) 26*8kB (UEC) 73*16kB (UEC) 15*32kB (UC) 7*64kB (UC) 2*128kB (U) 2*256kB (EC) 1*512kB (C) 1*1024kB (C) 1*2048kB (C) 113*4096kB (C) = 469516kB
Mar 16 15:16:46 ace kernel: lowmem_reserve[]: 0 0 7284 7284
Mar 16 15:16:46 ace kernel: DMA free:469848kB boost:0kB min:3140kB low:3924kB high:4708kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:3264kB inactive_file:88968kB unevictable:0kB writepending:0kB present:786432kB managed:626976kB mlocked:0kB bounce:0kB free_pcp:684kB local_pcp:88kB free_cma:466912kB
Mar 16 15:16:46 ace kernel: Node 0 active_anon:101904kB inactive_anon:860kB active_file:100316kB inactive_file:182184kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:68716kB dirty:164kB writeback:0kB shmem:800kB writeback_tmp:0kB kernel_stack:1768kB pagetables:2228kB sec_pagetables:0kB all_unreclaimable? yes
Mar 16 15:16:46 ace kernel: active_anon:25691 inactive_anon:215 isolated_anon:0
active_file:25079 inactive_file:45546 isolated_file:0
unevictable:0 dirty:41 writeback:0
slab_reclaimable:6994 slab_unreclaimable:5831
mapped:17179 shmem:200 pagetables:557
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:1905040 free_pcp:171 free_cma:116728
Mar 16 15:16:46 ace kernel: Mem-Info:
Mar 16 15:16:46 ace kernel: 1fe0: 00000000 bebf6040 00000000 b6b48494
Mar 16 15:16:46 ace kernel: 1fc0: 000a0802 02139c08 b6f65a20 00000142 00000000 00000000 bebf60d7 00000001
Mar 16 15:16:46 ace kernel: 1fa0: 000a0802 02139c08 ffffff9c 02139c08 000a0802 00000000
Mar 16 15:16:46 ace kernel: Exception stack(0xf0ad1fa8 to 0xf0ad1ff0)
Mar 16 15:16:46 ace kernel: sys_openat from __sys_trace_return+0x0/0x10
Mar 16 15:16:46 ace kernel: do_sys_openat2 from sys_openat+0x98/0xd4
Mar 16 15:16:46 ace kernel: do_filp_open from do_sys_openat2+0xbc/0x184
Mar 16 15:16:46 ace kernel: path_openat from do_filp_open+0x98/0x134
Mar 16 15:16:46 ace kernel: link_path_walk.part.0.constprop.0 from path_openat+0x94/0x111c
Mar 16 15:16:46 ace kernel: walk_component from link_path_walk.part.0.constprop.0+0x250/0x3a8
Mar 16 15:16:46 ace kernel: __lookup_slow from walk_component+0xf4/0x164
Mar 16 15:16:46 ace kernel: d_alloc_parallel from __lookup_slow+0x60/0x138
Mar 16 15:16:46 ace kernel: d_alloc from d_alloc_parallel+0x50/0x3b8
Mar 16 15:16:46 ace kernel: __d_alloc from d_alloc+0x18/0x74
Mar 16 15:16:46 ace kernel: kmem_cache_alloc_lru from __d_alloc+0x2c/0x1bc
Mar 16 15:16:46 ace kernel: ___slab_alloc from kmem_cache_alloc_lru+0x4fc/0x640
Mar 16 15:16:46 ace kernel: new_slab from ___slab_alloc+0x3e8/0xa0c
Mar 16 15:16:46 ace kernel: __alloc_pages from new_slab+0x384/0x43c
Mar 16 15:16:46 ace kernel: out_of_memory from __alloc_pages+0xa98/0x1044
Mar 16 15:16:46 ace kernel: oom_kill_process from out_of_memory+0x218/0x34c
Mar 16 15:16:46 ace kernel: dump_header from oom_kill_process+0x23c/0x248
Mar 16 15:16:46 ace kernel: dump_stack_lvl from dump_header+0x54/0x1fc
Mar 16 15:16:46 ace kernel: show_stack from dump_stack_lvl+0x90/0xac
Mar 16 15:16:46 ace kernel: unwind_backtrace from show_stack+0x18/0x1c
Mar 16 15:16:46 ace kernel: Hardware name: BCM2711
Mar 16 15:16:46 ace kernel: CPU: 3 PID: 203 Comm: systemd-journal Tainted: G C 6.1.19-1-rpi-ARCH #1
Mar 16 15:16:46 ace kernel: systemd-journal invoked oom-killer: gfp_mask=0xcd0(GFP_KERNEL|__GFP_RECLAIMABLE), order=0, oom_score_adj=-250
Mar 16 15:16:46 ace kernel: Out of memory: Killed process 1433 (rslsync) total-vm:334268kB, anon-rss:124684kB, file-rss:10408kB, shmem-rss:0kB, UID:1001 pgtables:476kB oom_score_adj:200
Mar 16 15:16:46 ace kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=rslsync,pid=1433,uid=1001
Mar 16 15:16:46 ace kernel: [ 1469] 1001 1469 227446 9429 184320 0 0 yay
Mar 16 15:16:46 ace kernel: [ 1455] 1001 1455 2198 969 40960 0 0 bash
Mar 16 15:16:46 ace kernel: [ 1453] 1001 1453 3216 889 40960 0 0 tmux: server
Mar 16 15:16:46 ace kernel: [ 1451] 1001 1451 3147 1000 45056 0 0 tmux: client
Mar 16 15:16:46 ace kernel: [ 1435] 1001 1435 2194 1002 40960 0 0 bash
Mar 16 15:16:46 ace kernel: [ 1434] 1001 1434 3549 1588 53248 0 0 sshd
Mar 16 15:16:46 ace kernel: [ 1433] 1001 1433 83567 33773 487424 0 200 rslsync
Mar 16 15:16:46 ace kernel: [ 1427] 1001 1427 10577 1226 77824 0 100 (sd-pam)
Mar 16 15:16:46 ace kernel: [ 1426] 1001 1426 4148 2312 57344 0 100 systemd
Mar 16 15:16:46 ace kernel: [ 1422] 0 1422 3509 2245 57344 0 0 sshd
Mar 16 15:16:46 ace kernel: [ 984] 971 984 10550 2542 94208 0 0 Plex Tuner Serv
Mar 16 15:16:46 ace kernel: [ 942] 971 942 11611 6795 98304 0 0 Plex Script Hos
Mar 16 15:16:46 ace kernel: [ 919] 971 919 33030 11109 315392 0 0 Plex Media Serv
Mar 16 15:16:46 ace kernel: [ 433] 33 433 2665 918 49152 0 0 nginx
Mar 16 15:16:46 ace kernel: [ 432] 33 432 2665 918 49152 0 0 nginx
Mar 16 15:16:46 ace kernel: [ 431] 33 431 2665 918 49152 0 0 nginx
Mar 16 15:16:46 ace kernel: [ 430] 33 430 2665 918 49152 0 0 nginx
Mar 16 15:16:46 ace kernel: [ 429] 0 429 2291 189 40960 0 0 nginx
Mar 16 15:16:46 ace kernel: [ 353] 0 353 1097 199 36864 0 0 agetty
Mar 16 15:16:46 ace kernel: [ 350] 0 350 2422 1624 45056 0 -1000 sshd
Mar 16 15:16:46 ace kernel: [ 347] 975 347 2942 130 40960 0 0 avahi-daemon
Mar 16 15:16:46 ace kernel: [ 346] 981 346 3874 1788 53248 0 0 systemd-network
Mar 16 15:16:46 ace kernel: [ 343] 0 343 3719 1635 57344 0 0 systemd-logind
Mar 16 15:16:46 ace kernel: [ 341] 81 341 2949 1252 49152 0 -900 dbus-daemon
Mar 16 15:16:46 ace kernel: [ 340] 975 340 3052 1163 53248 0 0 avahi-daemon
Mar 16 15:16:46 ace kernel: [ 319] 0 319 3351 426 32768 0 -1000 auditd
Mar 16 15:16:46 ace kernel: [ 317] 979 317 6075 1625 61440 0 0 systemd-timesyn
Mar 16 15:16:46 ace kernel: [ 316] 980 316 5147 2789 65536 0 0 systemd-resolve
Mar 16 15:16:46 ace kernel: [ 315] 0 315 12192 1058 73728 0 0 rngd
Mar 16 15:16:46 ace kernel: [ 216] 0 216 6984 1802 65536 0 -1000 systemd-udevd
Mar 16 15:16:46 ace kernel: [ 203] 0 203 15881 2065 86016 0 -250 systemd-journal
Mar 16 15:16:46 ace kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Mar 16 15:16:46 ace kernel: Tasks state (memory values in pages):
Mar 16 15:16:46 ace kernel: 131072 pages cma reserved
Mar 16 15:16:46 ace kernel: 39864 pages reserved
Mar 16 15:16:46 ace kernel: 1864704 pages HighMem/MovableOnly
Mar 16 15:16:46 ace kernel: 2061312 pages RAM
Mar 16 15:16:46 ace kernel: Total swap = 6143996kB
Mar 16 15:16:46 ace kernel: Free swap = 6143996kB
Mar 16 15:16:46 ace kernel: 0 pages in swap cache
Mar 16 15:16:46 ace kernel: 70759 total pagecache pages
Mar 16 15:16:46 ace kernel: DMA: 4*4kB (UC) 41*8kB (UEC) 73*16kB (UEC) 15*32kB (UC) 7*64kB (UC) 2*128kB (U) 2*256kB (EC) 1*512kB (C) 1*1024kB (C) 1*2048kB (C) 113*4096kB (C) = 469640kB
Mar 16 15:16:46 ace kernel: lowmem_reserve[]: 0 0 7284 7284
Mar 16 15:16:46 ace kernel: DMA free:469848kB boost:0kB min:3140kB low:3924kB high:4708kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:3264kB inactive_file:88968kB unevictable:0kB writepending:0kB present:786432kB managed:626976kB mlocked:0kB bounce:0kB free_pcp:572kB local_pcp:280kB free_cma:466912kB
Mar 16 15:16:46 ace kernel: Node 0 active_anon:209576kB inactive_anon:860kB active_file:100316kB inactive_file:182184kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:76284kB dirty:164kB writeback:0kB shmem:800kB writeback_tmp:0kB kernel_stack:1856kB pagetables:2572kB sec_pagetables:0kB all_unreclaimable? yes
Mar 16 15:16:46 ace kernel: active_anon:52394 inactive_anon:215 isolated_anon:0
active_file:25079 inactive_file:45546 isolated_file:0
unevictable:0 dirty:41 writeback:0
slab_reclaimable:6994 slab_unreclaimable:5831
mapped:19071 shmem:200 pagetables:643
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:1878958 free_pcp:143 free_cma:116728
Mar 16 15:16:46 ace kernel: Mem-Info:
Mar 16 15:16:46 ace kernel: dfe0: abff1a34 abff2a28 b6d47b58 b6d83868
Mar 16 15:16:46 ace kernel: dfc0: b2eb0464 b2eb0458 b2eb047c 00000142 abff2b90 abff2a84 00c57ea0 abff2c54
Mar 16 15:16:46 ace kernel: dfa0: b2eb0464 b2eb0458 ffffff9c b2e55cf0 000a4800 00000000
Mar 16 15:16:46 ace kernel: Exception stack(0xf0eedfa8 to 0xf0eedff0)
Mar 16 15:16:46 ace kernel: sys_openat from __sys_trace_return+0x0/0x10
Mar 16 15:16:46 ace kernel: do_sys_openat2 from sys_openat+0x98/0xd4
Mar 16 15:16:46 ace kernel: do_filp_open from do_sys_openat2+0xbc/0x184
Mar 16 15:16:46 ace kernel: path_openat from do_filp_open+0x98/0x134
Mar 16 15:16:46 ace kernel: link_path_walk.part.0.constprop.0 from path_openat+0x94/0x111c
Mar 16 15:16:46 ace kernel: walk_component from link_path_walk.part.0.constprop.0+0x250/0x3a8
Mar 16 15:16:46 ace kernel: __lookup_slow from walk_component+0xf4/0x164
Mar 16 15:16:46 ace kernel: d_alloc_parallel from __lookup_slow+0x60/0x138
Mar 16 15:16:46 ace kernel: d_alloc from d_alloc_parallel+0x50/0x3b8
Mar 16 15:16:46 ace kernel: __d_alloc from d_alloc+0x18/0x74
Mar 16 15:16:46 ace kernel: kmem_cache_alloc_lru from __d_alloc+0x2c/0x1bc
Mar 16 15:16:46 ace kernel: ___slab_alloc from kmem_cache_alloc_lru+0x4fc/0x640
Mar 16 15:16:46 ace kernel: new_slab from ___slab_alloc+0x3e8/0xa0c
Mar 16 15:16:46 ace kernel: __alloc_pages from new_slab+0x384/0x43c
Mar 16 15:16:46 ace kernel: out_of_memory from __alloc_pages+0xa98/0x1044
Mar 16 15:16:46 ace kernel: oom_kill_process from out_of_memory+0x218/0x34c
Mar 16 15:16:46 ace kernel: dump_header from oom_kill_process+0x23c/0x248
Mar 16 15:16:46 ace kernel: dump_stack_lvl from dump_header+0x54/0x1fc
Mar 16 15:16:46 ace kernel: show_stack from dump_stack_lvl+0x90/0xac
Mar 16 15:16:46 ace kernel: unwind_backtrace from show_stack+0x18/0x1c
Mar 16 15:16:46 ace kernel: Hardware name: BCM2711
Mar 16 15:16:46 ace kernel: CPU: 2 PID: 1500 Comm: rslsync Tainted: G C 6.1.19-1-rpi-ARCH #1
Mar 16 15:16:45 ace kernel: rslsync invoked oom-killer: gfp_mask=0xcd0(GFP_KERNEL|__GFP_RECLAIMABLE), order=0, oom_score_adj=200
')
uhhyeahbret
 
Posts: 12
Joined: Thu Sep 03, 2015 11:10 am

Re: Failed to disable unit: Invalid argument w/ systmctl dis

Postby gavinbeatty » Sat Mar 18, 2023 12:29 am

I'm currently dealing with the same issues. Downgrading kernel (I believe to 6.1.16) has not resolved the OOM, and I haven't been able to downgrade systemd (or rebuild the package with a backport fix) to attempt to resolve the disable issue.
gavinbeatty
 
Posts: 9
Joined: Fri Mar 17, 2023 9:54 pm

Re: Failed to disable unit: Invalid argument w/ systmctl dis

Postby gavinbeatty » Sat Mar 18, 2023 1:51 am

Disregarding OOM issue which is still ongoing for me but off-topic: I was able to get systemctl disable working again by rebuilding the systemd and systemd-libs packages with c63bfd0884cf20e48befbee49d41f667660a8802 added to the _backports array in PKGBUILD.
gavinbeatty
 
Posts: 9
Joined: Fri Mar 17, 2023 9:54 pm

Re: Random OOM errors and Invalid argument w/ systmctl disab

Postby gavinbeatty » Wed Mar 22, 2023 4:32 am

@uhhyeahbret are you still experiencing the OOM issue? I am, and am currently highly restricted in what I can run on my RPi as a result -- my only "workaround" has been to uninstall much of the software I was using. Even running a zsh shell with zimfw often causes OOM killer trigger, so I definitely had to uninstall the media server software I was using. Do you have any better workarounds, such as a downgrade of a specific package?

Also, do you have any insight into the possible cause? It seems to have happened in the last week.

PS: apologies that I said the OOM was off-topic, I somehow missed that it was in the subject since the start.
gavinbeatty
 
Posts: 9
Joined: Fri Mar 17, 2023 9:54 pm

Re: Random OOM errors and Invalid argument w/ systmctl disab

Postby graysky » Wed Mar 22, 2023 7:58 am

Re: OOM on armv7h, I posted the following in the OOM thread but I do not yet see that anyone has taken the time to report this upstream. Would one of you do it?

$this->bbcode_second_pass_quote('graysky', 'C')an someone open an issue upstream.
graysky
Developer
 
Posts: 1727
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000

Re: Random OOM errors and Invalid argument w/ systmctl disab

Postby gavinbeatty » Wed Mar 22, 2023 2:09 pm

Thanks for the reference, I hadn't found that.
gavinbeatty
 
Posts: 9
Joined: Fri Mar 17, 2023 9:54 pm

Re: Random OOM errors and Invalid argument w/ systmctl disab

Postby graysky » Sat Mar 25, 2023 11:15 am

Does 6.1.21-2 fix the issue?
graysky
Developer
 
Posts: 1727
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000


Return to User Questions

Who is online

Users browsing this forum: No registered users and 9 guests