From cc00c64e6610decd5cb87774e7e3511eb98b8aec Mon Sep 17 00:00:00 2001 From: Gerrit Date: Fri, 14 Jun 2024 14:24:39 +0200 Subject: [PATCH] Updates on MEP-4 (#70) --- docs/src/development/proposals/MEP4/README.md | 241 +++++++++++++----- 1 file changed, 178 insertions(+), 63 deletions(-) diff --git a/docs/src/development/proposals/MEP4/README.md b/docs/src/development/proposals/MEP4/README.md index f4c38bd6d1..221b1d1fbc 100644 --- a/docs/src/development/proposals/MEP4/README.md +++ b/docs/src/development/proposals/MEP4/README.md @@ -1,97 +1,212 @@ -# Multi-tenancy for the metal-api +# Multi-Tenancy for the metal-api -In the past we decided to treat the metal-api as a "low-level API", i.e. the API does not know anything about projects and tenants. A user with editor access can for example assign machines to every project he desires, he can see all the machines available and control them. Even though we always wanted to keep open the possibility to just offer bare metal machines to the end-user, the ultimate objective has always been to create an API for Kubernetes clusters. Hence, we tried to keep the metal-api code base as small as possible and we added resource scoping to a "higher-level API", the cloud-api, a component that is not open-source. From there, a user would be able to only see his own clusters and IP addresses. +!!! info -The implication is that the metal-api has no multi-tenancy without another layer on top of it that implements resource scoping. One can say that we treat clusters as first-class citizens. In regard of clusters we fulfill the objective that we had from the very beginning: provide a multi-tenant API for Kubernetes clusters to the end-users. + This document is work in progress. -However, as time passed by, things changed: The metal-stack is becoming an open-source product and we already have promising adopters of our product, who are willing to contribute to metal-stack. This is a serious chance of making our product better and more successful. It turns out that the decision we made was sufficient for us, but for others it is not. +In the past we decided to treat the metal-api as a "low-level API", i.e. the API does not specifically deal with projects and tenants. A user with editor access can for example assign machines to every project he desires, he can see all the machines available and can control them. We tried to keep the metal-api code base as small as possible and we added resource scoping to a "higher-level APIs". From there, a user would be able to only see his own clusters and IP addresses. -## Why adopters need multi-tenancy in the metal-api +As time passed metal-stack has become an open-source project and people are willing to adopt. Adopters who want to put their own technologies on top of the metal-stack infrastructure don't have those "higher-level APIs" that we implemented closed-source for our user base. So, external adopters most likely need to implement resource scoping on their own. -### Not every adopter will be interested in the cloud-api +Introducing multi-tenancy to the metal-api is a serious chance of making our product better and more successful as it opens the door for: -For example, users who want to combine the Metal Stack with Gardener, may not want to hide all of the Gardener's functionality behind the cloud-api in the way we do. They want to use the much more powerful Gardener Dashboard instead. The Gardener itself does not need the cloud-api either. It is a cluster-api by itself. It only needs to utilize our "low-level API" and actually expects this API to have multi-tenancy as otherwise every logged in user can create / destroy clusters in every existing project from the Gardener dashboard. +- Becoming a "fully-featured" API +- Narrowing down attack surfaces and possibility of unintended resource modification produced by bugs or human errors +- Discouraging people to implement their own scoping layers in front of the metal-stack +- Gaining performance through resource scopes +- Letting untrusted / third-parties work with the API -This makes obvious that, with our decision, we placed an unnecessary obstacle in the way of our adopters: They now need to implement an own layer between the Gardener and the metal-api to provide multi-tenancy. From the Gardener-perspective we strongly differ from other cloud providers in this aspect and it is a matter of time when this will become an issue. When we encourage adopters to implement such interfaces on their own we also partly lose control of our product, we increase divergence. +## Table of Contents -### We cannot claim that Metal Stack is a multi-tenant solution on our website +```@contents +Pages = ["README.md"] +Depth = 5 +``` -As the cloud-api is not part of the Metal Stack, the promise of multi-tenancy is only true for our network layer. Without the cloud-api to enable multi-tenancy, the network isolation is currently useless for end-users. Users of the Metal Stack can not self-manage machines, networks and ips without compromising the environment and thus, there is no self-service. We lose a valuable selling point when adopters can not immediately make use of our leading edge network isolation where we put so much effort to. +## Requirements -### Open partitions for third-party usage +These are some general requirements / higher objectives that MEP-4 has to fulfill. -If a third-party uses Gardener and our metal-api had multi-tenancy, we would be able to allow a third-party to create clusters with workers in our own partitions. At the moment, this is not possible because the Gardener needs to know the HMAC secrets to create worker nodes, which would compromise our environment. If a thirdy-party knows our HMAC we lose control over the machines of our own tenants. +- Should be able to run with mini-lab without requiring to setup complex auth backends (dex, LDAP, keycloak, ...) + - Simple to start with, more complex options for production setups +- Should utilize auth mechanisms that we have already in place to best possible degree +- Fine-grained access permissions (every endpoint maps to a permission) +- Tenant scoping (disallow resource access to resources of other tenants) +- Project scoping (disallow resource access to resources of other projects) +- Access tokens in self-service for technical user access -### We do not actually want to open-source the cloud-api +## Implementation -One could think about solving the multi-tenancy issue by adding machine endpoints to the cloud-api. Gardener would then not consume the metal-api anymore but only the cloud-api. +We gathered a lot of knowledge while implementing a multi-tenancy-capable backend for metalstack.cloud. The goal is now to use the same technology and adopt that to the metal-api, this includes: -This approach would not be ideal. We only want to offer a minimum viable product to adopters. The Gardener does not need a cluster-api as provided by the cloud-api. We want to treat additions on top of the basic stack as enterprise products. +- gRPC in combination with connectrpc +- OPA for making auth decisions +- REST HTTP only for OIDC login flows -The cloud-api contains billing endpoints, which are a perfect example for an optional addition of the Metal Stack. For basic usage of the Metal Stack a user does not need billing. Still, billing functionality can be interesting for some enterprises, who are like us, selling the infrastructure to third-parties. +### API Definitions -### Increased security for provider admins +The API definitions should be located on a separate Github repository separate from the server implementation. The proposed repository location is: https://github.com/metal-stack/api. -Multi-tenancy in the metal-api also has the potential to limit the damage that a provider administrator can cause by mistake. If an administrator has to acquire project permissions on machine-level we can effectively reduce the damage he can make to this single project. +This repository contains the `proto3` specification of the exposed metal-stack api. This includes the messages, simple validations, services and the access permission to these services. The input parameters for the authorization in the backend are generated from the `proto3` annotations. -Another example would be the automatic provisioning of a Gitlab CI runner used for integration testing (a use case that we have where we do not require the cloud-api). This can easily be done in automated manner with Ansible and the Metal dynamic inventory + modules. However, with Ansible, mistakes in the automation can be made very quickly and if Ansible would only see machines of a dedicated project, this would also reduce damage it can make. +Client implementations for the most relevant languages (go, python) are generated automatically. -It is likely that there are more similar use-cases like that to come (maybe even for the storage solution?). +This api is divided into end-user and admin access at the top level. The proposed APIs are: -Also the surface for our Gardener components (metal-ccm, gardener-extension-provider-metal, machine-controller-manager) would be reduced to project scopes. +- `api.v2`: For end-user facing services +- `admin.v2`: For operators and controllers which need access to unscoped entities -## Conclusion +The methods of the API can have different role scopes (and can be narrowed down further with fine-grained method permissions): -For these reasons the decision we made is very likely to have a negative impact on the adoption-rate of the Metal Stack and we should think about treating machines, networks and ips as first-class citizens as well. This makes us closer to the offer of hyperscalers. As mentioned in the beginning, all the time we tried to keep the possibility open to just offer bare metal machines. Let's continue with decision by adding multi-tenancy to the metal-api. +- `tenant`: Tenant-scoped methods, e.g. project creation (tenant needs to be provided in the request payload) + - Available roles: VIEWER, EDITOR, OWNER +- `project`: Project-scoped methods, e.g. machine creation (tenant needs to be provided in the request payload) + - Available roles: VIEWER, EDITOR, OWNER +- `admin` Admin-scoped methods, e.g. unscoped tenant list or switch register + - Available roles: VIEWER, EDITOR -## Required actions +And has methods with different visibility scopes: -### Resource scoping +- `self`: Methods that only the logged in user can access, e.g. show permissions with the presented token +- `public`: Methods that do not require any specific authorization +- `private`: Methods that are not exposed -Just as implemented by the cloud-api, resource scoping needs to be added to almost every endpoint of the metal-api: +### API -- Machines / Firewalls - - A user should only be able to view machines / firewalls of the projects he has at least view access to - - A user should only be able to create and destroy machines / firewalls for projects he has at least editor access to - Provider-tenants with at least view access can additionally view machines which have no project assignments - Provider-tenants with at least editor access can additionally allocate / reserve machines which have no project assignments -- Networks - - A user should only be able to view networks of the projects he has at least view access to - - A user should only be able to allocate networks of projects he has at least editor access to - - A user should only be able to free networks assigned to projects he has at least editor access to - Provider-tenants with at least view access can additionally view networks which have no project assignments - Provider-tenants with at least editor access can additionally edit networks which have no project assignments - Provider-tenants with at least admin access can additionally create or remove networks which have no project assignments -- IPs - - A user should only be able to view ips of the projects he has at least view access to - - A user should only be able to allocate ips in networks of projects he has at least editor access to - - A user should only be able to free ips assigned to projects he has at least editor access to -- Projects - - A logged in user is able to create projects when he has the permission to create projects - - A user should only be able to view projects where he has at least view access to - - A user should only be able to delete projects where he has admin access to -- Partitions / Images - - Only provider-admin users can add, delete, update - - All logged in users can view -- IPMI - - Only provider-tenants can view machine IPMI data -- Endpoints for internal use - - Should only be accessible with HMAC auth and the HMAC secrets are only known by components of the Metal Stack (mainly for communication between partition and control plane), never for third-party usage +The API server implements the services defined in the API and validates access to a method using OPA with the JWT tokens passed in the requests. The server is implemented using the connectrpc.com framework. -For all of this we need enhance the database queries with a filter for projects that a user has access to. As we already use a client to the masterdata-api in the metal-api, we can extract project memberships of a logged in user from there. +The API server implements the login flow through OIDC. After successful authentication, the API server derives user permissions from the OIDC provider and issues a new JWT token which is passed on to the user. The tokens including the permissions are stored in a redis compatible backend. -### More permissions +With these tokens, users can create Access Tokens for CI/CD or other use cases. -We do not only need `kaas-...` permissions in the LDAP but also `maas-`. This way we can differentiate between permissions for the cloud-api and permissions for the metal-api. +JWT Tokens can be revoked by admins and the user itself. -### Service account tokens / technical users +### API Server -We need to provide the possibility for users to obtain access tokens to use for technical purposes (CI, third-party tooling like Gardener, ...). +Is put into a new github repo which implements the services defined in the `api` repository. It opens a `https` endpoints where the grpc (via connectrpc.com) and oidc servives are exposed. -We do not have this functionality yet, but it would also become a necessity for the cloud-api at some point in the future. +### Migration of the Consumers -### Cloud API +To allow consumers to migrate to the `v2` API gradually, both apis, the new and the old, are deployed in parallel. In the control-plane both apis are deployed side-by-side behind the ingress. `api.example.com` is forwarded to `metal-api` and `metal.example.com` is forwarded to the new `api-server`. -- Project creation and deletion again have to be moved back into the metal-api, this also frees adopters from the need to write an own API in order to manage projects- The cloud-api will (again) only proxy project endpoints through to the metal-api -- Do not point the secret bindings to a the shared provider secret in a partition. Create an individual provider-secret for the logged in tenant. The Gardener needs to use this tenant-specific provider secret to talk to the metal-api, do not give the Gardener HMAC access anymore. -- The provider secret partition mapping can be removed from the cloud-api config and from the deployment +The api-server will talk to the existing metal-api during the process of migration services away to the new grpc api. + +The migration process can be done in the following manner: + +for each resource in the metal-api: + +- create a new proto3 based definition in the `api` repo. +- implement at least a small wrapper service in the `api-server` which asks the metal-api for this resource an maps the response back the caller in the grpc format. +- identify all consumers of this resource and replace them to use the grpc instead of the rest api +- move the business logic incl. the backend calls to ipam, metal-db, masterdata-ap, nsq for this resource from the metal-api to the api-server + +We will try to migrate the rethinkdb backend implementation to a generic approach during this effort. + +There are a lot of consumers of metal-api, which need to be migrated: + +- ansible +- firewall-controller +- firewall-controller-manager +- gardener-extension-auth +- gardener-extension-provider-metal + - Do not point the secret bindings to a the shared provider secret in the seed anymore. Instead, use individual provider-secret containing project-scoped API access tokens in the Gardener project namespaces. +- machine-controller-manager-provider-metal +- metal-ccm +- metal-console +- metal-bmc +- metal-core +- metal-hammer +- metal-image-cache-sync +- metal-images +- metal-metrics-exporter +- metal-networker +- metalctl +- pixie + +## User Scenarios + +This section gathers a collection of workflows from the perspective of a user that we want to provide with the implementation of this proposal. + +### Machine Creation + +A regular user wants to create a machine resource. + +Requirements: Project was created, permissions are present + +- The user can see networks that were provided by the admin. + + ``` + $ metalctl network ls + ID NAME PROJECT PARTITION NAT SHARED PREFIXES IPS + internet Internet Network true false 212.34.83.0/27  ● + tenant-super-network-fra-equ01 Project Super Network fra-equ01 false false 10.128.0.0/14  ● + underlay-fra-equ01 Underlay Network fra-equ01 false false 10.0.0.0/16  ● + ``` + +- The user has to set the project scope first or provide `--project` flags for all commands. + ``` + $ metalctl project set 793bb6cd-8b46-479d-9209-0fedca428fe1 + You are now acting on project 793bb6cd-8b46-479d-9209-0fedca428fe1. + ``` +- The user can create the child network required for machine allocation. + ``` + $ metalctl network allocate --partition fra-equ01 --name test + ``` +- Now, the user sees his own child network. + ``` + $ metalctl network ls + ID NAME PROJECT PARTITION NAT SHARED PREFIXES IPS + internet Internet Network true false 212.34.83.0/27  ● + tenant-super-network-fra-equ01 Project Super Network fra-equ01 false false 10.128.0.0/14  ● + └─╴08b9114b-ec47-4697-b402-a11421788dc6 test 793bb6cd-8b46-479d-9209-0fedca428fe1 fra-equ01 false false 10.128.64.0/22  ● + underlay-fra-equ01 Underlay Network fra-equ01 false false 10.0.0.0/16  ● + ``` +- The user does not see any machines yet. + ``` + $ metalctl machine ls + ``` +- The user can create a machine. + ``` + $ metalctl machine create --networks internet,08b9114b-ec47-4697-b402-a11421788dc6 --name test --hostname test --image ubuntu-20.04 --partition fra-equ01 --size c1-xlarge-x86` + ``` +- The machine will now be provisioned. + ``` + $ metalctl machine ls + ID LAST EVENT WHEN AGE HOSTNAME PROJECT SIZE IMAGE PARTITION + 00000000-0000-0000-0000-ac1f6b7befb2 Phoned Home 20s 50d 4h test 793bb6cd-8b46-479d-9209-0fedca428fe1 c1-xlarge-x86 Ubuntu 20.04 20210415 fra-equ01 + ``` + +!!! warning + + A user **cannot** list all allocated machines for all projects. The user **must** always switch project context first and can only view the machines inside this project. Only admins can see all machines at once. + +### Scopes for Resources + +The admins / operators of the metal-stack should be able to provide _global_ resources that users are able to use along with their own resources. In particular, users can view and use _global_ resources, but they are not allowed to create, modify or delete them. + +!!! info + + When a project ID field is empty on a resource, the resource is considered _global_. + +Where possible, users should be capable of creating their own resource entities. + +| Resource | User | Global | +| :----------------- | :--- | :----- | +| File System Layout | yes | yes | +| Firewall | yes | | +| Firmware | | yes | +| OS Image | | yes | +| Machine | yes | | +| Network (Base) | | yes | +| Network (Children) | yes | | +| IP | yes | | +| Partition | | yes | +| Project | yes | | +| Project Token | yes | | +| Size | | yes | +| Switch | | | +| Tenant | | yes | + +!!! info + + Example: A user can make use of the file system layouts provided by the admins, but can also create own layouts. Same applies for images. As soon as a user creates own resources, the user takes over the responsibility for the machine provisioning to succeed.