From ceb41c0d2ec917f99c06b546b2d18cbab3f8a301 Mon Sep 17 00:00:00 2001 From: Gabriele Filaferro Date: Mon, 1 Feb 2021 12:06:25 +0100 Subject: [PATCH] Added initial version of doc for tenant operator --- operators/Makefile | 2 +- operators/README.md | 119 ++++++++++++++++-- operators/api/v1alpha1/tenant_types.go | 20 +-- operators/api/v1alpha1/workspace_types.go | 3 + operators/cmd/tenant-operator/main.go | 6 +- .../crds/crownlabs.polito.it_tenants.yaml | 11 +- .../crds/crownlabs.polito.it_workspaces.yaml | 4 +- 7 files changed, 139 insertions(+), 26 deletions(-) diff --git a/operators/Makefile b/operators/Makefile index 9a762b294..7fbdf4c7f 100644 --- a/operators/Makefile +++ b/operators/Makefile @@ -52,7 +52,7 @@ CONTROLLER_GEN=$(shell which controller-gen) endif run-tenant: generate fmt vet manifests - go run cmd/tenant-operator/main.go\ + go run cmd/tenant-operator/main.go\ --target-label=reconcile=true\ --kc-url=$(KEYCLOAK_URL)\ --kc-tenant-operator-user=$(KEYCLOAK_TENANT_OPERATOR_USER)\ diff --git a/operators/README.md b/operators/README.md index f721c25ea..9fc21c475 100644 --- a/operators/README.md +++ b/operators/README.md @@ -55,17 +55,19 @@ helm upgrade crownlabs-instance-operator deploy/instance-operator \ Based on [Kubebuilder 2.3](https://github.com/kubernetes-sigs/kubebuilder.git), the operator implements the environment creation logic of CrownLabs. -Upon the creation of a *Instance*, the operator triggers the creation of the following components: -* Kubevirt VirtualMachine Instance and the logic to access the noVNC instance inside the VM (Service, Ingress) -* An instance of [Oauth2 Proxy](https://github.com/oauth2-proxy/oauth2-proxy) (Deployment, Service, Ingress) to regulate access to the VM. +Upon the creation of a _Instance_, the operator triggers the creation of the following components: + +- Kubevirt VirtualMachine Instance and the logic to access the noVNC instance inside the VM (Service, Ingress) +- An instance of [Oauth2 Proxy](https://github.com/oauth2-proxy/oauth2-proxy) (Deployment, Service, Ingress) to regulate access to the VM. All those resources are bound to the Instance life-cycle via the [OwnerRef property](https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/) ### APIs/CRDs The Instance Operator implements the backend logic necessary to spawn new environments starting from a predefined template. The operator relies on two Kubernetes Custom Resource Definitions (CRDs) which implement the basic APIs: -* **Template** defines the size of the execution environment (e.g.; Virtual Machine), its base image and a description. This object is created by managers and read by users, while creating new instances. -* **Instance** defines an instance of a certain template. The manipulation of those objects triggers the reconciliation logic in the operator, which creates/destroy associated resources (e.g.; Virtual Machines). + +- **Template** defines the size of the execution environment (e.g.; Virtual Machine), its base image and a description. This object is created by managers and read by users, while creating new instances. +- **Instance** defines an instance of a certain template. The manipulation of those objects triggers the reconciliation logic in the operator, which creates/destroy associated resources (e.g.; Virtual Machines). ### Build from source @@ -77,15 +79,15 @@ go build ./cmd/instance-operator/main.go #### Testing -N.B. So far, the readiness check for VirtualMachines is performed by assuming that the operator is running on the same cluster of the Virtual Machines. This prevents the possibility to have *ready* VMs when testing the operator outside the cluster. +N.B. So far, the readiness check for VirtualMachines is performed by assuming that the operator is running on the same cluster of the Virtual Machines. This prevents the possibility to have _ready_ VMs when testing the operator outside the cluster. ## SSH bastion The SSH bastion is composed of two basic blocks: + 1. `bastion-operator`: an operator based on on [Kubebuilder 2.3](https://github.com/kubernetes-sigs/kubebuilder.git) 2. `ssh-bastion`: a lightweight alpine based container running [sshd](https://linux.die.net/man/8/sshd) - #### SSH Key generation In order to deploy the SSH bastion, it is necessary to generate in advance the keys that will be used by the ssh daemon. @@ -93,6 +95,7 @@ These keys are automatically generated by an helm hook in case not already prese Nonetheless, for reference purposes, the following is the manual procedure achieving the same goal: 1. Generate the host keys needed to run sshd using: + ```bash # Generate the keys in this folder (they will be ignored by git) or in a folder outside the project ssh-keygen -f ssh_host_key_ecdsa -N "" -t ecdsa -C "" @@ -101,6 +104,7 @@ ssh-keygen -f ssh_host_key_rsa -N "" -t rsa -C "" ``` 2. Create the secret holding the keys: + ```bash kubectl create secret generic \ --namespace: @@ -109,6 +113,107 @@ kubectl create secret generic \ --from-file=./ssh_host_key_rsa ``` +## Tenant operator + +The tenant operator manages users inside the Crownlabs cluster, its workflow is based upon 2 CRDs: + +- `Tenant`: represents a CrownLabs user (i.e. a student or a professor), describing the basic contact information and the Workspaces he is granted access to. +- `Workspace`: represents the collection of related `Templates` (e.g. those belonging to the same course) + +### Main actions performed by the operator + +The operator executes some actions to setup cluster and external resources to allow users to operate properly. + +- To maintain consistency inside and outside the cluster, actions are executed + - after every update on the `Tenant` or `Workspace` resource + - after every update on the cluster resources managed by the operator itself. + - periodically, (i.e. in a range between 1 and 2 hours since the last update) +- When performing the actions, the operator utilizes a `fail-fast:false` strategy. Hence, if an action fails (e.g. due to Keycloak being temporarily offline), the operator does not stop and tries to execute all the other independent actions. + +The actions performed by the operator are the following: + +- `Tenant` ([details](pkg/tenant-controller/tenant_controller.go)) + - update the tenant resource with labels for firstName and lastName + - create or update some cluster resources: + - namespace: to host all other cluster resources related to the tenant + - resourceQuota; to limit resources used by the tenant + - roleBindings: to allow user to manage own instances + - clusterRole: to allow user to watch own resource + - clusterRoleBinding: to bind the above cluster role + - networkPolicies: + - one to allow to send/receive traffic to own instances + - one to deny send/receive traffic to instances of other users + - check if the tenant subscribed to non-existing workspaces, in case some are found add to the tenant status + - append a label for each subscribed workspace that exists + - create or update the corresponding user in keycloak and assign him/her a role for each subscribed workspace + - create or update the nextcloud credentials for the user + - delete all managed resources upon tenant deletion +- `Workspace` ([details](pkg/tenant-controller/workspace_controller.go)) + - create or update some cluster resources + - namespace: to host all other cluster resources related to the workspace + - clusterRoleBinding: to allow managers of the workspace to interact with all instances inside the workspace + - roleBindings + - one to allow users inside the workspace to view the available templates + - one to allow managers of the workspace to edit templates of the workspace + - create the corresponding keycloak roles to allow tenant to consume them + - delete all managed resources upon workspace deletion + - upon deletion, unsubscribe all tenants which previously subscribed to the workspace + +### Usage + +``` +go run cmd/tenant-operator/main.go + --target-label=reconcile=true\ + --kc-url=KEYCLOAK_URL\ + --kc-tenant-operator-user=KEYCLOAK_TENANT_OPERATOR_USER\ + --kc-tenant-operator-psw=KEYCLOAK_TENANT_OPERATOR_PSW\ + --kc-login-realm=KEYCLOAK_LOGIN_REALM\ + --kc-target-realm=KEYCLOAK_TARGET_REALM\ + --kc-target-client=KEYCLOAK_TARGET_CLIENT\ + --nc-url=NEXTCLOUD_URL\ + --nc-tenant-operator-user=NEXTCLOUD_TENANT_OPERATOR_USER\ + --nc-tenant-operator-psw=NEXTCLOUD_TENANT_OPERATOR_PSW + + +Arguments: + --target-label + The key=value pair label that needs to be in the resource to be reconciled. A single pair in the format key=value + --kc-url + The URL of the keycloak server + --kc-tenant-operator-user + The username of the acting account for keycloak + --kc-tenant-operator-psw + The password of the acting account for keycloak + --kc-login-realm + The realm where to login the keycloak acting account + --kc-target-realm + The target realm for keycloak clients, roles and users + --kc-target-client + The target client for keycloak users and roles + --nc-url + The URL of the nextcloud server + --nc-tenant-operator-user + The username of the acting account for nextcloud + --nc-tenant-operator-psw + The password of the acting account for nextcloud +``` + +For local development (e.g. using [KinD](https://kind.sigs.k8s.io/)), the operator can be easily started using `make`, after having set the proper environment variables regarding the different configurations: + +```make +make install-tenant +make run-tenant +``` + +### CRD definitions + +For a deeper definition go to + +- `Tenant` [GoLang code version](./api/v1alpha1/tenant_types.go) +- `Tenant` [YAML version](./deploy/crds/crownlabs.polito.it_tenants.yaml) +- `Workspace` [GoLang code version](./api/v1alpha1/workspace_types.go) +- `Workspace` [YAML version](./deploy/crds/crownlabs.polito.it_workspaces.yaml) + ## CrownLabs Image List The CrownLabs Image List script allows to to gather the list of available images from a Docker Registry and expose it as an ImageList custom resource, to be consumed from the CrownLabs dashboard. diff --git a/operators/api/v1alpha1/tenant_types.go b/operators/api/v1alpha1/tenant_types.go index 9fbaf009e..f5ca84f1e 100644 --- a/operators/api/v1alpha1/tenant_types.go +++ b/operators/api/v1alpha1/tenant_types.go @@ -23,7 +23,6 @@ import ( // EDIT THIS FILE! THIS IS SCAFFOLDING FOR YOU TO OWN! // NOTE: json tags are required. Any new fields you add must have json tags for the fields to be serialized. -// WorkspaceUserRole is an enum for the role of a user in a workspace // +kubebuilder:validation:Enum=manager;user type WorkspaceUserRole string @@ -36,14 +35,15 @@ const ( // UserWorkspaceData contains the info of the workspaces related to a user type UserWorkspaceData struct { - WorkspaceRef GenericRef `json:"workspaceRef"` - GroupNumber uint `json:"groupNumber,omitempty"` - Role WorkspaceUserRole `json:"role"` + // reference to the workspace resource in the cluster + WorkspaceRef GenericRef `json:"workspaceRef"` + GroupNumber uint `json:"groupNumber,omitempty"` + // role of the user in the context of the workspace + Role WorkspaceUserRole `json:"role"` } // TenantSpec defines the desired state of Tenant type TenantSpec struct { - // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster // Important: Run "make" to regenerate code after modifying this file FirstName string `json:"firstName"` @@ -53,10 +53,10 @@ type TenantSpec struct { // +kubebuilder:validation:Pattern="^[a-zA-Z0-9.!#$%&'*+\\/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$" Email string `json:"email"` - // list of workspaces the user is subscribed to + // list of workspace the tenant subscribed to Workspaces []UserWorkspaceData `json:"workspaces,omitempty"` - // public keys of tenant + // list of the public keys of the tenant for SSH access PublicKeys []string `json:"publicKeys,omitempty"` // should the resource create the sandbox namespace for k8s practice environment @@ -66,9 +66,9 @@ type TenantSpec struct { // TenantStatus defines the observed state of Tenant type TenantStatus struct { - // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster // Important: Run "make" to regenerate code after modifying this file + // info about the namespace for the user resources inside the cluster PersonalNamespace NameCreated `json:"personalNamespace"` SandboxNamespace NameCreated `json:"sandboxNamespace"` @@ -77,7 +77,9 @@ type TenantStatus struct { // list of subscriptions to non-k8s services (keycloak, nextcloud, ..) Subscriptions map[string]SubscriptionStatus `json:"subscriptions"` - Ready bool `json:"ready"` + + // false if there have been errors within the last reconcile, true otherwise + Ready bool `json:"ready"` } // +kubebuilder:object:root=true diff --git a/operators/api/v1alpha1/workspace_types.go b/operators/api/v1alpha1/workspace_types.go index c9a46d025..a63f1600a 100644 --- a/operators/api/v1alpha1/workspace_types.go +++ b/operators/api/v1alpha1/workspace_types.go @@ -28,6 +28,7 @@ type WorkspaceSpec struct { // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster // Important: Run "make" to regenerate code after modifying this file + // human-readable name of the workspace PrettyName string `json:"prettyName"` } @@ -36,11 +37,13 @@ type WorkspaceStatus struct { // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster // Important: Run "make" to regenerate code after modifying this file + // info about the namespace for the workspace resources inside the cluster Namespace NameCreated `json:"namespace,omitempty"` // list of subscriptions to non-k8s services (keycloak, nextcloud, ..) Subscriptions map[string]SubscriptionStatus `json:"subscription,omitempty"` + // false if there have been errors within the last reconcile, true otherwise Ready bool `json:"ready,omitempty"` } diff --git a/operators/cmd/tenant-operator/main.go b/operators/cmd/tenant-operator/main.go index 10b47372b..4cc46da7c 100644 --- a/operators/cmd/tenant-operator/main.go +++ b/operators/cmd/tenant-operator/main.go @@ -67,11 +67,11 @@ func main() { "Enable leader election for controller manager. "+ "Enabling this will ensure there is only one active controller manager.") flag.StringVar(&targetLabel, "target-label", "", "The key=value pair label that needs to be in the resource to be reconciled. A single pair in the format key=value") - flag.StringVar(&kcURL, "kc-url", "", "The URL of the keycloak client.") + flag.StringVar(&kcURL, "kc-url", "", "The URL of the keycloak server.") flag.StringVar(&kcTnOpUser, "kc-tenant-operator-user", "", "The username of the acting account for keycloak.") flag.StringVar(&kcTnOpPsw, "kc-tenant-operator-psw", "", "The password of the acting account for keycloak.") - flag.StringVar(&kcLoginRealm, "kc-login-realm", "", "The realm where to login the keycloak account.") - flag.StringVar(&kcTargetRealm, "kc-target-realm", "", "The target realm for keycloak clients and roles.") + flag.StringVar(&kcLoginRealm, "kc-login-realm", "", "The realm where to login the keycloak acting account.") + flag.StringVar(&kcTargetRealm, "kc-target-realm", "", "The target realm for keycloak clients, roles and users.") flag.StringVar(&kcTargetClient, "kc-target-client", "", "The target client for keycloak users and roles.") flag.StringVar(&ncURL, "nc-url", "", "The base URL for the nextcloud actor.") flag.StringVar(&ncTnOpUser, "nc-tenant-operator-user", "", "The username of the acting account for nextcloud.") diff --git a/operators/deploy/crds/crownlabs.polito.it_tenants.yaml b/operators/deploy/crds/crownlabs.polito.it_tenants.yaml index 8e33e027b..f6782599d 100644 --- a/operators/deploy/crds/crownlabs.polito.it_tenants.yaml +++ b/operators/deploy/crds/crownlabs.polito.it_tenants.yaml @@ -60,25 +60,25 @@ spec: lastName: type: string publicKeys: - description: public keys of tenant + description: list of the public keys of the tenant for SSH access items: type: string type: array workspaces: - description: list of workspaces the user is subscribed to + description: list of workspace the tenant subscribed to items: description: UserWorkspaceData contains the info of the workspaces related to a user properties: groupNumber: type: integer role: - description: WorkspaceUserRole is an enum for the role of a user in a workspace + description: role of the user in the context of the workspace enum: - manager - user type: string workspaceRef: - description: GenericRef stores generric data to point to a kubernetes resource + description: reference to the workspace resource in the cluster properties: name: type: string @@ -106,7 +106,7 @@ spec: type: string type: array personalNamespace: - description: NameCreated contains info about the status of a resource + description: info about the namespace for the user resources inside the cluster properties: created: type: boolean @@ -116,6 +116,7 @@ spec: - created type: object ready: + description: false if there have been errors within the last reconcile, true otherwise type: boolean sandboxNamespace: description: NameCreated contains info about the status of a resource diff --git a/operators/deploy/crds/crownlabs.polito.it_workspaces.yaml b/operators/deploy/crds/crownlabs.polito.it_workspaces.yaml index 40e4375f1..b4d6613f3 100644 --- a/operators/deploy/crds/crownlabs.polito.it_workspaces.yaml +++ b/operators/deploy/crds/crownlabs.polito.it_workspaces.yaml @@ -43,6 +43,7 @@ spec: description: WorkspaceSpec defines the desired state of Workspace properties: prettyName: + description: human-readable name of the workspace type: string required: - prettyName @@ -51,7 +52,7 @@ spec: description: WorkspaceStatus defines the observed state of Workspace properties: namespace: - description: NameCreated contains info about the status of a resource + description: info about the namespace for the workspace resources inside the cluster properties: created: type: boolean @@ -61,6 +62,7 @@ spec: - created type: object ready: + description: false if there have been errors within the last reconcile, true otherwise type: boolean subscription: additionalProperties: