run ceph-csi as non-root and configurable UID/GID

[adabuleanu]

Describe the feature you'd like to have

Run ceph-csi as non-root and with configurable UID/GID.

What is the value to the end user? (why is it a priority?)

In an enterprise environment, running containers as root is a security concern.
For example, Ceph daemons support running as non-root with configurable UID/GID
https://docs.ceph.com/en/pacific/man/8/ceph/?highlight=setuser#cmdoption-ceph-setuser

--setuser user
will apply the appropriate user ownership to the file specified by the option ‘-o’.

--setgroup group
will apply the appropriate group ownership to the file specified by the option ‘-o’.

If ceph csi would support running as non-root with configurable UID/GID, the solution would be complete.

How will we know we have a good solution? (acceptance criteria)

Check OS process owner. Example for ceph-mgr

$ ps aux
167       244582  1.3  2.3 1547924 387256 ?      Ssl  Sep15  23:30 ceph-mgr --fsid=4b8c33bf-eaa1-45fa-a350-c40b17c04e2a --keyring=/etc/ceph/keyring-store/keyring --log-to-stderr=true --err-to-stderr=true --mon-cluster-log-to-stderr=true --log-stderr-prefix=debug  --default-log-to-file=false --default-mon-cluster-log-to-file=false --mon-host=[v2:10.10.10.7:3300,v1:10.10.10.7:6789],[v2:10.10.10.172:3300,v1:10.10.10.172:6789],[v2:10.10.10.112:3300,v1:10.10.10.112:6789] --mon-initial-members=a,b,c --id=a --setuser=ceph --setgroup=ceph --client-mount-uid=0 --client-mount-gid=0 --foreground

Additional context

This feature was mentioned in #714, but the ticket was closed due to inactivity.

[Madhu-1]

@ceph/ceph-csi-contributors any volunteers for analysis/PR to check is it possible especially for daemonset pods?

[nixpanic]

Mounting will still require CAP_SYSADMIN, so the DaemonSet Pods can not drop a lot of privileges. The provisioner and other components should not need to run as root.

[Madhu-1]

To add to above statement we cannot run the daemonset plugin pod with normal users as we need to create/delete the csi socket on the hostPath and the hostPath is created by kubelet with root user and if we try to run the daemonset plugin pod with the normal user the plugin pod will never come up

$ ls -lrt
total 36
drwxr-xr-x  2 root root 4096 Sep 28 10:34 pki
-rw-------  1 root root   62 Sep 28 10:34 cpu_manager_state
-rw-r--r--  1 root root  189 Sep 28 10:35 kubeadm-flags.env
-rw-r--r--  1 root root  944 Sep 28 10:35 config.yaml
drwxr-x---  2 root root 4096 Sep 28 10:35 pod-resources
drwxr-x---  2 root root 4096 Sep 29 07:56 plugins_registry
drwxr-x---  5 root root 4096 Sep 29 12:34 plugins
drwxr-xr-x  2 root root 4096 Sep 29 12:51 device-plugins
drwxr-x--- 24 root root 4096 Sep 29 12:51 pods


drwxr-xr-x 2 root root 4096 Sep 29 12:34 rook-ceph.rbd.csi.ceph.com
drwxr-xr-x 2 root root 4096 Sep 29 12:34 rook-ceph.cephfs.csi.ceph.com


E0929 12:57:17.589273       1 node_register.go:42] failed to stat the socket /registration/rook-ceph.cephfs.csi.ceph.com-reg.sock with error: stat /registration/rook-ceph.cephfs.csi.ceph.com-reg.sock: permission denied


F0929 13:02:07.734019       1 server.go:130] Failed to listen: listen unix //csi/csi.sock: bind: permission denied
the user &{Uid:0 Gid:0 Username:root Name:root HomeDir:/root} and error <nil>


the group &{Gid:0 Name:root} and error <nil>
F0929 13:03:23.706695       1 server.go:124] Failed to remove //csi/csi.sock, error: remove //csi/csi.sock: permission denied

[leseb]

To add to above statement we cannot run the daemonset plugin pod with normal users as we need to create/delete the csi socket on the hostPath and the hostPath is created by kubelet with root user and if we try to run the daemonset plugin pod with the normal user the plugin pod will never come up

$ ls -lrt
total 36
drwxr-xr-x  2 root root 4096 Sep 28 10:34 pki
-rw-------  1 root root   62 Sep 28 10:34 cpu_manager_state
-rw-r--r--  1 root root  189 Sep 28 10:35 kubeadm-flags.env
-rw-r--r--  1 root root  944 Sep 28 10:35 config.yaml
drwxr-x---  2 root root 4096 Sep 28 10:35 pod-resources
drwxr-x---  2 root root 4096 Sep 29 07:56 plugins_registry
drwxr-x---  5 root root 4096 Sep 29 12:34 plugins
drwxr-xr-x  2 root root 4096 Sep 29 12:51 device-plugins
drwxr-x--- 24 root root 4096 Sep 29 12:51 pods


drwxr-xr-x 2 root root 4096 Sep 29 12:34 rook-ceph.rbd.csi.ceph.com
drwxr-xr-x 2 root root 4096 Sep 29 12:34 rook-ceph.cephfs.csi.ceph.com


E0929 12:57:17.589273       1 node_register.go:42] failed to stat the socket /registration/rook-ceph.cephfs.csi.ceph.com-reg.sock with error: stat /registration/rook-ceph.cephfs.csi.ceph.com-reg.sock: permission denied


F0929 13:02:07.734019       1 server.go:130] Failed to listen: listen unix //csi/csi.sock: bind: permission denied
the user &{Uid:0 Gid:0 Username:root Name:root HomeDir:/root} and error <nil>


the group &{Gid:0 Name:root} and error <nil>
F0929 13:03:23.706695       1 server.go:124] Failed to remove //csi/csi.sock, error: remove //csi/csi.sock: permission denied

How about using an init container to chown the directory first and then run under the 167 UID?

[gurucloudsec]

Is this still slated for support in 3.12?

[Madhu-1]

@gurucloudsec not for 3.12 may be for the next release. adding a flag as we are looking for volunteer for this one.

[clayrisser]

I am very interested in this feature. For context, I use the following storageclass parameters with aws efs. I'm hoping something equivalent will be supported on cephfs.

  parameters:
    directoryPerms: "700"
    gid: "1000"
    gidRangeEnd: "2000"
    gidRangeStart: "1000"
    uid: "1000"