linux-rpi version 6.12.1-1 not enabling cgroups [fixed]

[viky]

Hello,
I'd love tho get help regarding new rpi kernel.
On my RPi4 I recently got an update to 6.12.1-1 of linux-rpi package and after reboot, k3s seems unable to see cgroups, outputting:

```
Nov 29 16:10:47 rpi4-64-2 k3s[1267]: time="2024-11-29T16:10:47+01:00" level=fatal msg="failed to find memory cgroup (v2)"
Nov 29 16:10:47 rpi4-64-2 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Nov 29 16:10:47 rpi4-64-2 systemd[1]: k3s.service: Failed with result 'exit-code'.
Nov 29 16:10:47 rpi4-64-2 systemd[1]: Failed to start Lightweight Kubernetes.
```

cmdline should be set up correctly:

```
$ cat /boot/cmdline.txt
root=/dev/sda2 rw rootwait console=serial0,115200 console=tty1 fsck.repair=yes usb-storage.quirks=152d:0576:u cgroup_memory=1 cgroup_enable=memory
```

/proc/cgroups is still outputting the same thing:
```
$ cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 0 45 1
cpu 0 45 1
cpuacct 0 45 1
blkio 0 45 1
devices 0 45 1
freezer 0 45 1
net_cls 0 45 1
perf_event 0 45 1
net_prio 0 45 1
pids 0 45 1
```

I also tried to copy the cmdline.txt into /boot/firmware/ as that's where it is in RaspiOS, but it makes no difference.

Downgrading to linux-rpi-6.6.62-1 fixes everything, but that's of course not a permanent solution.

Any input is welcomed
Thanks

[graysky]

Upstream changes with 6.12 deprecated legacy cgroup v1 memory controller. Is k3s expecting this? Please try 6.12.1-2 when it hits the repos.

[squall0gd]

Hello,
I've hit into this as well, so I've tried to debug to determine what is going on. The problem is that memory cgroup controller is disabled and there is no way to re-enable it for 6.12(or at least, I do not see such way). Apparently `cgroup_enable` kernel command line parameter is missing for 6.12, while memory cgroups are disabled with kernel cmdline with `cgroup_disable` defined in device tree binaries for RPI. You can verify this with simple `cat /proc/cmdline`.

```
reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe cgroup_disable=memory numa_policy=interleave iommu_dma_numa_policy=interleave system_heap.max_order=0 smsc95xx.macaddr=2C:CF:67:34:85:9A vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000 root=/dev/sda2 rw rootwait console=ttyAMA10,115200 console=tty1 fsck.repair=yes cgroup_memory=1 cgroup_enable=memory
```

- commit which has introduced cgroup_disable parameter in dtb:make_clickable_callback(MAGIC_URL_FULL, ' ', 'https://github.com/raspberrypi/linux/commit/86099deff5abf5f63643eecaedb4c11ae77474ce.', '', ' class="postlink"') DTB source code has been copied into 6.12.y tag
- cgroup.c file for 6.6.y, which has `cgroup_enable`:make_clickable_callback(MAGIC_URL_FULL, ' ', 'https://github.com/raspberrypi/linux/blob/rpi-6.6.y/kernel/cgroup/cgroup.c#L6754-L6777', '', ' class="postlink"')
- cgroup.c file for 6.12.y, which has not `cgroup_enable`:https://github.com/raspberrypi/linux/blob/rpi-6.12.y/kernel/cgroup/cgroup.c#L6846-L6856[/url]

[viky]

Hello, thank you both.
Sadly the 6.12.1-2 version makes no difference.
Also k3s seems to be requiring cgroups v2 (and not supporting v1 at all [url]https://docs.k3s.io/advanced#known-issues-with-rootless-mode[/url] maybe I'm reading it the wrong way, but cgroups v2 is supported).
I think @squall0gd nailed it, making it an upstream issue :/

[eric-dev]

EDIT:
@squall0gd or @viky - did you end up opening an upstream issue?

I'm noticing the same thing, the memory controller appears to be missing.

using this kernel: `6.12.1-3-rpi`

```
➜ ~ doas ls /sys/fs/cgroup | grep memory
memory.pressure
```
```
➜ ~ cat /sys/fs/cgroup/cgroup.controllers
cpuset cpu io pids
```

```
➜ ~ cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 0 56 1
cpu 0 56 1
cpuacct 0 56 1
blkio 0 56 1
devices 0 56 1
freezer 0 56 1
net_cls 0 56 1
perf_event 0 56 1
net_prio 0 56 1
pids 0 56 1
```

[viky]

Hello, I did not open an upstream issue yet. I'm guessing this one is related [url]https://github.com/raspberrypi/linux/issues/6396[/url]. But it talks about cgroup v1, so maybe it's not a fix. I'm a bit lost in it, to be honest :/

[squall0gd]

Hey, I have not opened any upstream issue @eric-dev. I've got pretty busy time at this moment, so for home lab I've just reverted back to 6.6.x.

[denk]

This might be solved bymake_clickable_callback(MAGIC_URL_FULL, ' ', 'https://github.com/raspberrypi/linux/pull/6524.', '', ' class="postlink"') Can't test right now, sorry. If I can, I will give feedback.

[denk]

[quote]This might be solved bymake_clickable_callback(MAGIC_URL_FULL, ' ', 'https://github.com/raspberrypi/linux/pull/6524.', '', ' class="postlink"') Can't test right now, sorry. If I can, I will give feedback.[/quote]

Thanks to graysky having incorporated the patch, I can confirm that the cgroup memory controller is enabled with cgroup_enable=memory in the cmdline.

[viky]

Hello,
version `6.12.3-2` fixed the issue for me. Thanks to everyone involved!
V