Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CPDEV-102250] Remove Cloud Provider Plugin support #675

Merged
merged 1 commit into from
Jun 24, 2024
Merged

[CPDEV-102250] Remove Cloud Provider Plugin support #675

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
168 changes: 0 additions & 168 deletions documentation/Installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@ This section provides information about the inventory, features, and steps for i
- [services](#services)
- [kubeadm](#kubeadm)
- [Kubernetes version](#kubernetes-version)
- [Cloud Provider Plugin](#cloud-provider-plugin)
- [Service Account Issuer](#service-account-issuer)
- [kubeadm_kubelet](#kubeadm_kubelet)
- [kubeadm_kube-proxy](#kubeadm_kube-proxy)
Expand Down Expand Up @@ -1180,173 +1179,6 @@ services:
kubernetesVersion: v1.26.11
```

#### Cloud Provider Plugin

Before proceeding further, it is recommended to read the official Kubernetes Guide about the CPP deployment in the cluster at [https://kubernetes.io/blog/2020/02/07/deploying-external-openstack-cloud-provider-with-kubeadm/](https://kubernetes.io/blog/2020/02/07/deploying-external-openstack-cloud-provider-with-kubeadm/).

**Warning**: Manual CPP installation on a deployed cluster can cause Kubernetes out-of-service denial and break Kubemarine procedures for adding and removing nodes.

It is possible to specify a plugin at the installation stage, if it is required. To enable the CPP support, just specify the `external-cloud-volume-plugin` parameter of `controllerManager` in the `kubeadm` cluster configuration. For example:

```yaml
services:
kubeadm:
controllerManager:
extraArgs:
external-cloud-volume-plugin: openstack
extraVolumes:
- name: "cloud-config"
hostPath: "/etc/kubernetes/cloud-config"
mountPath: "/etc/kubernetes/cloud-config"
readOnly: true
pathType: File
```

In this case, Kubemarine automatically initializes and joins new cluster nodes with CPP enabled. However, this is not enough for the full operation of the CPP. There are a number of manual steps required to configure the CPP before running Calico and other plugins. These steps depend directly on your Cloud Provider and its specific settings. An example of a simple setup for an openstack is as follows:

1. Prepare cloud config of your Cloud Provider with credentials and mandatory parameters required for the connection. Openstack cloud config example:

```ini
[Global]
region=RegionOne
username=username
password=password
auth-url=https://openstack.cloud:5000/v3
tenant-id=14ba698c0aec4fd6b7dc8c310f664009
domain-name=default
```

1. Upload the cloud config to all the nodes in the cluster to the following location:

```
/etc/kubernetes/cloud-config
```

It is recommended to use Kubemarine functionality of plugins or thirdparties for automatic uploading. For example, it is possible to upload the cloud config on all nodes using thirdparties before starting the cluster installation:

```yaml
services:
thirdparties:
/etc/kubernetes/cloud-config:
source: ./example/cloud-config.txt
```

1. Before running any plugins, it is necessary to create a secret RBAC resource and cloud controller manager DaemonSet for CPP. This can be specified as the very first Kubemarine plugin, for example:

Create a file `./openstack-cloud-controller-manager-ds.yaml` on deploy node with the following content:

```shell
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: cloud-controller-manager
namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: openstack-cloud-controller-manager
namespace: kube-system
labels:
k8s-app: openstack-cloud-controller-manager
spec:
selector:
matchLabels:
k8s-app: openstack-cloud-controller-manager
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
k8s-app: openstack-cloud-controller-manager
spec:
nodeSelector:
node-role.kubernetes.io/control-plane: ""
tolerations:
- key: node.cloudprovider.kubernetes.io/uninitialized
value: "true"
effect: NoSchedule
- key: node-role.kubernetes.io/control-plane
effect: NoSchedule
- effect: NoSchedule
key: node.kubernetes.io/not-ready
serviceAccountName: cloud-controller-manager
containers:
- name: openstack-cloud-controller-manager
image: docker.io/k8scloudprovider/openstack-cloud-controller-manager:v1.15.0
securityContext:
privileged: true
args:
- /bin/openstack-cloud-controller-manager
- --v=1
- --cloud-config=$(CLOUD_CONFIG)
- --cloud-provider=openstack
- --use-service-account-credentials=true
- --address=127.0.0.1
volumeMounts:
- mountPath: /etc/kubernetes/pki
name: k8s-certs
readOnly: true
- mountPath: /etc/config
name: cloud-config-volume
readOnly: true
- mountPath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
name: flexvolume-dir
resources:
requests:
cpu: 200m
env:
- name: CLOUD_CONFIG
value: /etc/config/cloud.conf
hostNetwork: true
volumes:
- hostPath:
path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
type: DirectoryOrCreate
name: flexvolume-dir
- hostPath:
path: /etc/kubernetes/pki
type: DirectoryOrCreate
name: k8s-certs
- name: cloud-config-volume
secret:
secretName: cloud-config
- name: ca-cert
secret:
secretName: openstack-ca-cert
```
**Warning:** Pay attention on external resources links.
For restricted environments links should be changed to local registry.
For example, image: `docker.io/k8scloudprovider/openstack-cloud-controller-manager:v1.15.0` should be changed to
`registry:17001/k8scloudprovider/openstack-cloud-controller-manager:v1.15.0`

**Warning**: Pay attention to pod security policies for cloud controller manager. You can create new ClusterRole or disable PSP.

Place the following plugin section to the cluster config:

```yaml
plugins:
cloud-config:
install: true
installation:
priority: -1
procedures:
- shell:
command: sudo kubectl create secret -n kube-system generic cloud-config --from-literal=cloud.conf="$(sudo cat /etc/kubernetes/cloud-config)" --dry-run -o yaml > cloud-config-secret.yaml && sudo kubectl apply -f cloud-config-secret.yaml
nodes: ['control-plane-1']
- shell:
command: sudo kubectl apply -f https://github.com/kubernetes/cloud-provider-openstack/raw/release-1.15/cluster/addons/rbac/cloud-controller-manager-roles.yaml
nodes: ['control-plane-1']
- shell:
command: sudo kubectl apply -f https://github.com/kubernetes/cloud-provider-openstack/raw/release-1.15/cluster/addons/rbac/cloud-controller-manager-role-bindings.yaml
nodes: ['control-plane-1']
- template:
source: ./openstack-cloud-controller-manager-ds.yaml
```
**Warning**: Pay attention on external resources links.
For restricted environments configs should be downloaded and links changed to the local path.

### Service Account Issuer

**Warning**:
Expand Down
7 changes: 0 additions & 7 deletions kubemarine/kubernetes/components.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -262,13 +262,6 @@ def get_init_config(cluster: KubernetesCluster, group: AbstractGroup[RunResult],
}
}

if inventory['services']['kubeadm']['controllerManager']['extraArgs'].get('external-cloud-volume-plugin'):
init_config['nodeRegistration'] = {
'kubeletExtraArgs': {
'cloud-provider': 'external'
}
}

if control_plane and worker:
init_config.setdefault('nodeRegistration', {})['taints'] = []

Expand Down
Loading