Document VolumeGroupSnapshots

[sftim]

This is a Feature Request

What would you like to be added
Update the volume snapshot page to explain VolumeGroupSnapshots

Why is this needed
VolumeGroupSnapshots are beta and we barely document them.

Comments
See kubernetes/enhancements#3476

Linking to the CSI documentation is not enough (but we should to that too).

/sig storage

[dipesh-rawat]

/language en
/triage accepted
/priority backlog

[dipesh-rawat]

Some useful links to help with this task:

• v1.32 Feature Blog on Volume Group Snapshots Moving To Beta: https://kubernetes.io/blog/2024/12/18/kubernetes-1-32-volume-group-snapshot-beta/
• CSI Volume Group Snapshot documentation: https://kubernetes-csi.github.io/docs/group-snapshot-restore-feature.html

[network-charles]

I wonder which driver was used to test this VolumeGroupSnapshots CRD. I tried it using EBS on an EKS cluster and it keeps telling me that the CRD isn't installed, but it is. AWS EBS was listed among the list of supported drivers.

groupsnapshotclass-v1beta1.yaml

apiVersion: groupsnapshot.storage.k8s.io/v1beta1
kind: VolumeGroupSnapshotClass
metadata:
  name: csi-ebs-groupsnapclass
driver: ebs.csi.aws.com
deletionPolicy: Delete

kubectl apply -f groupsnapshotclass-v1beta1.yaml

error: resource mapping not found for name: "csi-ebs-groupsnapclass" namespace: "" from "groupsnapshotclass-v1beta1.yaml": no matches for kind "VolumeGroupSnapshotClass" in version "groupsnapshot.storage.k8s.io/v1beta1"
ensure CRDs are installed first

kubectl get crd

NAME                                                       CREATED AT
cninodes.vpcresources.k8s.aws                              2025-01-06T19:11:50Z
eniconfigs.crd.k8s.amazonaws.com                           2025-01-06T19:12:49Z
policyendpoints.networking.k8s.aws                         2025-01-06T19:11:50Z
securitygrouppolicies.vpcresources.k8s.aws                 2025-01-06T19:11:50Z
volumegroupsnapshotclasses.groupsnapshot.storage.k8s.io    2025-01-06T19:31:03Z
volumegroupsnapshotcontents.groupsnapshot.storage.k8s.io   2025-01-06T19:31:03Z
volumegroupsnapshots.groupsnapshot.storage.k8s.io          2025-01-06T19:31:04Z
volumesnapshotclasses.snapshot.storage.k8s.io              2025-01-06T19:31:04Z
volumesnapshotcontents.snapshot.storage.k8s.io             2025-01-06T19:31:04Z
volumesnapshots.snapshot.storage.k8s.io                    2025-01-06T19:31:04Z

When I tried it using the csi-hostpath-driver on a minikube cluster, it accepted the CRD but I wasn't getting a list of generated snapshots as instructed here.
I expected something similar to the below.

kubectl get volumegroupsnapshot new-groupsnapshot-demo  -o yaml | sed -n '/pvcVolumeSnapshotRefList/,$p'

  pvcVolumeSnapshotRefList:
  - persistentVolumeClaimRef:
      name: hpvc
    volumeSnapshotRef:
      name: snapshot-4bcc4a322a473abf32babe3df5779d14349542b1f0eb6f9dab0466a85c59cd42-2024-06-19-12.35.17
  - persistentVolumeClaimRef:
      name: hpvc-2
    volumeSnapshotRef:
      name: snapshot-62bd0be591e1e10c22d51748cd4a53c0ae8bf52fabb482bee7bc51f8ff9d9589-2024-06-19-12.35.17
  readyToUse: true

[Madhu-1]

@network-charles I sent a PR to update the existing document to use beta API kubernetes-csi/docs#612. AFAIK EBS doesn't support VGS yet, it's supported in a few other drivers like RBD,cephFS, etc.

[network-charles]

Thanks for opening the PR, @Madhu-1. I’ll try those drivers.

[Madhu-1]

@network-charles you could try with host-path as well. with above changes in the PR. Not that you need to update the host-path sts with below

...
- --feature-gates=CSIVolumeGroupSnapshot=true
image: registry.k8s.io/sig-storage/csi-snapshotter:v8.2.0

[network-charles]

What cluster do you run it on?

I ran mine on minikube using the below commands.

# Start minikube
minikube start
minikube addons enable csi-hostpath-driver
minikube addons enable volumesnapshots

# Install VolumeGroupSnapshot CRD
git clone https://github.com/kubernetes-csi/external-snapshotter.git
cd external-snapshotter
kubectl kustomize client/config/crd | kubectl create -f -

[Madhu-1]

What cluster do you run it on?

I ran mine on minikube using the below commands.

# Start minikube
minikube start
minikube addons enable csi-hostpath-driver
minikube addons enable volumesnapshots

# Install VolumeGroupSnapshot CRD
git clone https://github.com/kubernetes-csi/external-snapshotter.git
cd external-snapshotter
kubectl kustomize client/config/crd | kubectl create -f -

it did the manual installation using https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/deploy/kubernetes-1.27/deploy.sh

[network-charles]

Ok, it works. How did you manually install the snapshot CRD?

[Madhu-1]

Ok, it works. How did you manually install the snapshot CRD?

yes with kubectl create i installed snapshot controller and VS and VGS CRD's

[network-charles]

Oh, in the external-snapshotter repo, right?

[Madhu-1]

Oh, in the external-snapshotter repo, right?

yes correct from the external-snapshotter repo

[network-charles]

@network-charles you could try with host-path as well. with above changes in the PR. Not that you need to update the host-path sts with below

...
- --feature-gates=CSIVolumeGroupSnapshot=true
image: registry.k8s.io/sig-storage/csi-snapshotter:v8.2.0

Nice.
It means you didn’t install the CSI snapshotter like you outlined here.

[Madhu-1]

@network-charles you could try with host-path as well. with above changes in the PR. Not that you need to update the host-path sts with below

...
- --feature-gates=CSIVolumeGroupSnapshot=true
image: registry.k8s.io/sig-storage/csi-snapshotter:v8.2.0

Nice. It means you didn’t install the CSI snapshotter like you outlined here.

The above is for the host-path (because the template in the current hostpath repo doesn't use 8.2.0 and also doesn't enable the feature-gate required for the vgs) not for the csi snapshot controller (if this is what you mean csi snapshotter)

[network-charles]

Oh, you mean it is in the csi-host-path repo. What folder in the repo is the statefulset file located?

[Madhu-1]

Oh, you mean it is in the csi-host-path repo. What folder in the repo is the statefulset file located?

https://github.com/kubernetes-csi/csi-driver-host-path/blob/245a8ee724d30c1575c34231ce086a4d6542a370/deploy/kubernetes-1.27/hostpath/csi-hostpath-plugin.yaml#L360 here it is

[network-charles]

Ah, I see. So it has to be updated to the below, correct?

        - name: csi-snapshotter
          image: registry.k8s.io/sig-storage/csi-snapshotter:v8.2.0
          args:
            - -v=5
            - --csi-address=/csi/csi.sock
            - --feature-gates=CSIVolumeGroupSnapshot=true

[network-charles]

Only the volume snapshot worked. The volume group snapshot didn't work on a kind and minikube cluster.

[Madhu-1]

i tried and it worked for me, have you enable the VGS featugate on the snapshot controller as well as mentioned in https://github.com/kubernetes-csi/external-snapshotter?tab=readme-ov-file#volume-group-snapshot-support

steps will be as below

• Install VGS beta v1 CRD
• use external-snapshot 8.2.0 in both hostpath and the csi-snapshotter controller
• update the hostpath and the csi-snapshotter controller (sts/deployment) to set the featuregate as mentioned here
• create VGSClass and VGS as mentioned in the csi pods

[network-charles]

I missed that. It works now, thank you.

[network-charles]

Ah, I see. So it has to be updated to the below, correct?

        - name: csi-snapshotter
          image: registry.k8s.io/sig-storage/csi-snapshotter:v8.2.0
          args:
            - -v=5
            - --csi-address=/csi/csi.sock
            - --feature-gates=CSIVolumeGroupSnapshot=true

We may need to open a PR specifying this update on the csi-driver-host-path docs. Or we could add it to the statefulset directly. I won't advise us to specify it here on the kubernetes documentation since it's an independent project. What do you think, @Madhu-1?

On the page, @sftim suggested, we can specify the VolumeGroupSnapshots and contents.

[Madhu-1]

We may need to open a PR specifying this update on the csi-driver-host-path docs. Or we could add it to the statefulset directly. I won't advise us to specify it here on the kubernetes documentation since it's an independent project. What do you think, @Madhu-1?

we don't need to add it to kube document, it will be added to the host-path and also its already documented in the external-snapshotter Readme.

cc @xing-yang