No more memory cgroups / raspberry pi kernel 4.9.51

Problems with packages? Post here, using [tags] of the package name.

No more memory cgroups / raspberry pi kernel 4.9.51

Postby pea » Fri Sep 29, 2017 9:58 am

Hi,

I'm using Docker on my raspberry pi3 and i noticed that cgroups are missing since the last kernel updates (i can't say the exact version).
Here is my system version:
+ Linux docker0 4.9.51-1-ARCH #1 SMP Sat Sep 23 00:35:12 UTC 2017 armv7l GNU/Linux
+ 32 bits
+ docker : 17.07.0-ce
All packages are up2date.

The cpuset group was already missing since many months. But now, the memory cgroup is missing too:

dockerd[338]: time="2017-09-26T06:53:50.229252098+02:00" level=warning msg="Your kernel does not support memory limit capabilities or the cgroup is not mounted. Limitation discarded."

Any ideas why it has changed ?

Regards,
pea
 
Posts: 5
Joined: Thu Mar 30, 2017 1:49 pm

Re: No more memory cgroups / raspberry pi kernel 4.9.51

Postby lehmanju » Fri Oct 06, 2017 1:37 pm

I can confirm this weird behavior for kernel "4.9.52". Kernel config shows its enabled but /proc/cgroups says its disabled. Even after adding "cgroup_enable=memory" to /boot/cmdline.txt it stays disabled.
lehmanju
 
Posts: 3
Joined: Fri Oct 06, 2017 1:29 pm

Re: No more memory cgroups / raspberry pi kernel 4.9.51

Postby graysky » Fri Oct 06, 2017 6:59 pm

@op - can you post the output of:
Code: Select all
% grep -i cgroup /proc/filesystems
nodev   cgroup
nodev   cgroup2


That is from my RPi3 running armv7h. It is identical to the output of my x86_64 box. Perhaps there is something more diagnostic?

EDIT: Does this help?

https://raw.githubusercontent.com/moby/ ... -config.sh
graysky
 
Posts: 356
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000

Re: No more memory cgroups / raspberry pi kernel 4.9.51

Postby lehmanju » Sat Oct 07, 2017 9:46 am

It is exactly the same for me, the problem is not that cgroups don't work in general, just the memory part of it seems to have stopped working for archlinux, even after adding "cgroup_enable=memory" to /boot/cmdline.txt.

I have also checked the script, but there is nothing wrong as far as I can tell:
Code: Select all
info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CFQ_GROUP_IOSCHED: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: missing
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled (as module)
  - "ipvlan":
    - CONFIG_IPVLAN: enabled (as module)
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled (as module)
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: enabled
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000
lehmanju
 
Posts: 3
Joined: Fri Oct 06, 2017 1:29 pm

Re: No more memory cgroups / raspberry pi kernel 4.9.51

Postby graysky » Sat Oct 07, 2017 10:20 am

I don't use docker but I do use lxc on my RPi3. My containers work as expected. The lxc package ships with a util to verify compatibility (lxc-checkconfig). Does docker have something similar?
graysky
 
Posts: 356
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000

Re: No more memory cgroups / raspberry pi kernel 4.9.51

Postby lehmanju » Sat Oct 07, 2017 6:24 pm

As far as I know, no, docker has no such utility. Nevertheless, it notifies you if something doesn't work.

As lxc uses cgroups as well, lxc-checkconfig should work for docker.

Here is my output:
Code: Select all
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled

--- Control groups ---
Cgroups: enabled

Cgroup v1 mount points:
/sys/fs/cgroup/systemd
/sys/fs/cgroup/blkio
/sys/fs/cgroup/net_cls,net_prio
/sys/fs/cgroup/freezer
/sys/fs/cgroup/perf_event
/sys/fs/cgroup/pids
/sys/fs/cgroup/cpu,cpuacct
/sys/fs/cgroup/cpuset
/sys/fs/cgroup/devices

Cgroup v2 mount points:
/sys/fs/cgroup/unified

Cgroup v1 clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, not loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, loaded
Advanced netfilter: enabled, not loaded
CONFIG_NF_NAT_IPV4: enabled, loaded
CONFIG_NF_NAT_IPV6: enabled, not loaded
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loadedCONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded
FUSE (for use with lxcfs): enabled, not loaded

--- Checkpoint/Restore ---
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: missing
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: missing
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig


Looks fine, it says memory controller enabled but /proc/cgroups says its disabled:
Code: Select all
#subsys_name    hierarchy       num_cgroups     enabled
cpuset  8       1       1
cpu     7       62      1
cpuacct 7       62      1
blkio   2       62      1
memory  0       85      0
devices 9       62      1
freezer 4       1       1
net_cls 3       1       1
perf_event      5       1       1
net_prio        3       1       1
pids    6       68      1


lxc-checkconfig only checks kernel configuration but not runtime configuration. There is a problem with enabling the memory controller, I also tried mounting it manually which shouldn't be necessary, usually.
lehmanju
 
Posts: 3
Joined: Fri Oct 06, 2017 1:29 pm

Re: No more memory cgroups / raspberry pi kernel 4.9.51

Postby lopsided » Sun Oct 22, 2017 3:50 am

Did anyone manage to find a solution to this? I have the exact same issue, but I'm trying to use systemd-nspawn.

I even tried manually mounting the memory cgroup, but I just get this error:
Code: Select all
mount: /sys/fs/cgroup/memory: special device none does not exist.
lopsided
 
Posts: 2
Joined: Sun Oct 22, 2017 3:44 am

Re: No more memory cgroups / raspberry pi kernel 4.9.51

Postby lopsided » Sun Oct 22, 2017 4:06 am

Actually, I just figured it out. This problem was introduced by this commit. You now need to add cgroup_memory=1 to the command line to enable memory cgroups.

Libvirt containers now work, but there is still a (possibly unrelated) problem with systemd-nspawn.
lopsided
 
Posts: 2
Joined: Sun Oct 22, 2017 3:44 am


Return to Packages

Who is online

Users browsing this forum: No registered users and 1 guest