Kubevirt Community Stack

Create Kubevirt VMs via Helm
for use with ArgoCD, Argo Workflows, KEDA, ClusterAPI, Tekton etc...

This project is in an alpha state. It is functional but missing automated tests and is only partially documented.
Some chart schemas may still change as more features are integrated.

Who is this for:

The Kubevirt-Community-Stack may be of interest if you:

• operate one or more physical computers which you would like to split into smaller virtual machiens.
• are already running kubernetes to orchestrate container workloads
• are already in the ArgoCD or Tekton ecosystem and/or work primarily with some other Helm-based tooling.
• want/need fully-featured VMs for things like hardware emulation, hardware-passthrough, Virtual Desktops, vGPU, etc... which are not suppoted by Micro-VMs such as Firecracker.
• want to integrate Kubevirt into your existing infrastructure without needing to adopt a full platform like OpenShift Virtuazation, HarvesterHCI, StarlingX, or KubeSphere etc...
• want to install and operate Kubevirt on an existing system withhout needing to re-image it with an installer ISO.

Component charts

Kubervirt

Kubevirt is a Kubernetes Virtualization API and runtime which controls QEMU/KVM virtual machine instances and provides the CRDs that define them. It's distrubuted as a Kubernetes Operator which is install via the kubevirt chart.

Kubevirt CDI

The Containerized Data Importer can pull virtual machine images, ISO files, and other types of bootable media from sources like S3, HTTP, or OCI images. This data is then written to PVCs which are mounted as disks. For examples of various ways to use the CDI, see the notes in Argocd-Apps

Cloud-Init

The Cloud-init helm chart allows the user to define the specification of a linux-based vm's operating system as code. In addition to basec cloud-init functions, his chart provides some extra functionality via an initjob that makes cloud-init more GitOps friendly.

Additional Features:

• Regex values using existing secrets or environmental variables via envsubst
• Create random user passwords or use an existing secret
• Download files from a URL
• Base64 encode + gzip your write_files content
• Populate Wireguard configuration values from an existsing secret
• Track the total size of user-data and check file for valid syntax

Kubevirt VM

The Kubevirt-VM Chart allows a user to easily template a Kubevirt VirtualMachine or VirtualMachinePool and its associated resources sudch as Disks, DataVolumes, Horizontal Pod Autoscaler, Network Policies, Service, Ingres, Probes, and Cloud-init data (via bundled cloud-init subchart).

Kubevirt Manager

This is a community-developed web-ui which allows users to create, manage, and interact with virtual machines running in Kubevirt. See their official docs at kubevirt-manager.io

Cluster API Operator & Addons

Cluster API provides a standardised kubernetes-native interface for creating k8s clusters using a wide variety of providers. The combined chart can install the Cluster API Operator as well as bootstrap the Cluster API Kubevirt Provider which allows creating k8s clusters from the CLI or as YAML using Kubevirt VMs. Cluster-api-provider-kubevirt also includes cloud-provider-kubevirt which enables the exposeure of LoadBalancer type services within tenant clusters to the host cluster. This negates the need for a dedicated loadbalancer such as MetalLB inside the tenant cluster.

See CAPI.md for a basic walkthrough of creating a CAPI-based tenant cluster.

CAPI Cluster

The CAPI Cluster helm chart provides a way to create workload clusters using the Kubevirt infrastructure, Kubeadm Bootstrap + ControlPlane, and Helm providers.

Dependencies

libvirt-clients

This utility will audit a host machine and report what virtualisation capabilities are available

• Installation

  sudo apt-get install -y libvirt-clients

• Usage

  $ virt-host-validate qemu
  QEMU: Checking for hardware virtualization          : PASS
  QEMU: Checking if device /dev/kvm exists            : PASS
  QEMU: Checking if device /dev/kvm is accessible     : PASS
  QEMU: Checking if device /dev/vhost-net exists      : PASS
  QEMU: Checking if device /dev/net/tun exists        : PASS

virtctl

virtctl is the command-line utility for managing Kubevirt resources. It can be installed as a standalone CLI or as a Kubectl plugin via krew.

• Standalone

  export VERSION=v0.41.0
  wget https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/virtctl-${VERSION}-linux-amd64

• Plugin

  kubectl krew install virt

clusterctl

The clusterctl CLI tool handles the lifecycle of a Cluster API management cluster.

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.7.2/clusterctl-linux-amd64 -o clusterctl
sudo install -o root -g root -m 0755 clusterctl /usr/local/bin/clusterctl

Install The Kubevirt-Community-Stack

• Install the combined chart (kubevirt-stack).

   helm repo add kubevirt https://cloudymax.github.io/kubevirt-community-stack
   helm install kubevirt-stack kubevirt/kubevirt-stack \
     --namespace kubevirt \
     --create-namespace

Expand to see individual chart installation

• kubevirt: Installs the Kubevirt Operator.

  helm repo add kubevirt https://cloudymax.github.io/kubevirt-community-stack
  helm install kubevirt kubevirt/kubevirt \
    --namespace kubevirt \
    --create-namespace

• Cluster API Operator: Installs the Cluster API Operator.

  Work in progress.

• kubevirt-cdi: Install the Containerized Data Importer.

  helm repo add kubevirt https://cloudymax.github.io/kubevirt-community-stack
  helm install kubevirt-cdi kubevirt/kubevirt-cdi \
    --namespace cdi \
    --create-namespace

• kubevirt-manager: Deploy the Kubevirt-Manager UI

  # Customize your own values.yaml before deploying
  helm repo add kubevirt https://cloudymax.github.io/kubevirt-charts
  helm install kubevirt-manager kubevirt/kubevirt-manager \
    --fnamespace kubevirt-manager \
    --create-namespace

Examples

Example documentation is a work in progress.

• Work with Disk Images via the CDI
• Create your first linux VM
• Deploy a VirtialMachinePool and scale it using KEDA (ToDo)
• Add a GPU to your VM (ToDo)
• Deploy a CAPI Cluster
• Create a Windows VM (ToDo)
• Capture and deploy system images using Clonezilla Live (ToDo)
• Deploying via ArgoCD
• Integrate with Argo Workflows/Events

Uninstall

In the event that Kubevirt does not uninstall gracefully, you may need to perform the following steps:

export RELEASE=v0.17.0

# --wait=true should anyway be default
kubectl delete -n kubevirt kubevirt kubevirt --wait=true

# this needs to be deleted to avoid stuck terminating namespaces
kubectl delete apiservices v1.subresources.kubevirt.io

# not blocking but would be left over
kubectl delete mutatingwebhookconfigurations virt-api-mutator

# not blocking but would be left over
kubectl delete validatingwebhookconfigurations virt-operator-validator

# not blocking but would be left over
kubectl delete validatingwebhookconfigurations virt-api-validator

kubectl delete -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-operator.yaml --wait=false

# Find hanging resources
kubectl api-resources --verbs=list --namespaced -o name   | xargs -n 1 kubectl get --show-kind --ignore-not-found -n kubevirt

# If namespace is stuck
kubectl get namespace "kubevirt" -o json   | tr -d "\n" | sed "s/\"finalizers\": \[[^]]\+\]/\"finalizers\": []/"   | kubectl replace --raw /api/v1/namespaces/kubevirt/finalize -f -