cgroups support in flux-core w/ RHEL8

[chu11]

With PR #4255, cgroups support via the libsdprocess library (launching processes via systemd) is supported. However, specific configuration with RHEL8 would be required.

1. The AllowedCPUs (i.e. cpusets) support is only supported with cgroupsv2 (i.e. unified cgroups) per the systemd.resource-control manpage. So unified cgroups needs to be enabled.

2. By default, it appears RHEL8 sets up cgroups delegation only for Memory and Tasks. We need to add for CPUs too. I believe that can be done by setting up:

> cat /etc/systemd/system/[email protected]/delegate.conf
[Service]
Delegate=cpu cpuset memory pids

3. There appears to be a bug in RHEL8's systemd (systemd-239-51.el8_5.3.x86_64 as of this writing) where cgroups are not always delegated to user services correctly. I believe all of the following issues point to the same issue

systemd/systemd#9763
systemd/systemd#9578
systemd/systemd#9512
systemd/systemd#9502

and the fix is (I think):

systemd/systemd#9502

which is in systemd v240 (I think).

Other than cgroups not working, the way I noticed this not working was by doing (note that 8556 is my UID, replace with relevant UID of user service):

> systemctl show [email protected] | grep MemoryCurrent
MemoryCurrent=[not set]

Normally, Memory should be set. e.g.

> systemctl show [email protected] | grep MemoryCurrent
MemoryCurrent=8548352

Another way to see this is to check if cgroup controllers for the service are set or not. e.g.

>cat /sys/fs/cgroup/user.slice/user-8556.slice/cgroup.controllers

if things are not working, the file will be empty. If things are working, this should be populated.

>cat /sys/fs/cgroup/user.slice/user-8556.slice/cgroup.controllers
cpuset io memory pids

To workaround this I tried out the instructions here:

https://unix.stackexchange.com/questions/624428/cgroups-v2-cgroup-controllers-not-delegated-to-non-privileged-users-on-centos-s

which I believe will populate all user services with the same delegations. We could instead set things up per user, which is what I think we should do:

> cat /etc/systemd/system/user-8556.slice
[Unit]
Before=systemd-logind.service

[Slice]
Slice=user.slice
MemoryAccounting=yes
CPUAccounting=yes
TasksAccounting=yes

[Install]
WantedBy=multi-user.target

TL;DR

recommend setting:

1. unified cgroups in kernel boot cmdline

> cat /etc/systemd/system/[email protected]/delegate.conf
[Service]
Delegate=cpu cpuset memory pids

> cat /etc/systemd/system/user-8556.slice
[Unit]
Before=systemd-logind.service

[Slice]
Slice=user.slice
MemoryAccounting=yes
CPUAccounting=yes
TasksAccounting=yes

[Install]
WantedBy=multi-user.target