Version 2.10.0 of the LINSTOR® Operator for Kubernetes, developed by LINBIT®, introduces support for volume group snapshots, a feature that enables crash-consistent, point-in-time snapshots of multiple persistent volumes simultaneously. Kubernetes 1.36 promoted the VolumeGroupSnapshot API to General Availability (GA) under the groupsnapshot.storage.k8s.io/v1 API group.
Before volume group snapshots, a Kubernetes operator could only take snapshots of one persistent volume at a time. For stateful applications that distribute their data across multiple PVCs (for example, a database that separates data from write-ahead logs, or a web application that stores relational data and user-uploaded media in separate volumes), this meant accepting a window of inconsistency between snapshots. Volume group snapshots close that window by instructing the CSI driver to snapshot all grouped volumes atomically, at the exact same point in time.
This article will guide you through setting up, testing, and verifying the volume group snapshot feature by using LINSTOR in Kubernetes.
Prerequisites
To follow the instructions in this article, you will need:
- A running Kubernetes cluster, version 1.36 or later.
- LINSTOR Operator version 2.10.0 or later deployed in your cluster. If you do not yet have LINSTOR in your cluster, you can follow the Integrating LINBIT® SDS with Kubernetes, a Quick Start Guide.
- LINSTOR storage backed by LVM thin-provisioned or ZFS thin-provisioned pools, which are required for snapshot support.
- CRDs from the CSI external-snapshotter project, and
snapshot-controllerdeployment, version 8.x. The steps in the next section cover installing these if you do not yet have them.
Enabling volume group snapshot support
Enabling volume group snapshots requires a one-time configuration change across three places: the CSI snapshotter CRDs, the snapshot-controller deployment, and the LINSTOR Operator’s CSI controller.
Installing the CSI snapshotter CRDs
The group snapshot CRDs ship alongside the standard snapshot CRDs in the external-snapshotter project. If you have already installed the CRDs by using the external-snapshotter repository, your deployment likely has the group snapshot CRDs already.
To install or update the snapshot CRDs, including the group snapshot CRDs, run the following command:
kubectl apply -k https://github.com/kubernetes-csi/external-snapshotter//client/config/crdAfter applying the CRDs, verify that the group snapshot resources are available:
kubectl api-resources --api-group=groupsnapshot.storage.k8s.ioThe output should list the three group snapshot resources:
NAME SHORTNAMES APIVERSION NAMESPACED KIND
volumegroupsnapshotclasses vgsclass,vgsclasses groupsnapshot.storage.k8s.io/v1 false VolumeGroupSnapshotClass
volumegroupsnapshotcontents vgsc,vgscs groupsnapshot.storage.k8s.io/v1 false VolumeGroupSnapshotContent
volumegroupsnapshots vgs groupsnapshot.storage.k8s.io/v1 true VolumeGroupSnapshotDeploying the snapshot-controller
If you have not already deployed the snapshot-controller, install it from the external-snapshotter project:
kubectl apply -k https://github.com/kubernetes-csi/external-snapshotter//deploy/kubernetes/snapshot-controllerAfter applying, verify that the snapshot-controller pods are running:
kubectl get pods -n kube-system -l app.kubernetes.io/name=snapshot-controllerNAME READY STATUS RESTARTS AGE
snapshot-controller-58b6fc7b8f-7vc9q 1/1 Running 0 15s
snapshot-controller-58b6fc7b8f-hdjjk 1/1 Running 0 15sEnabling the CSIVolumeGroupSnapshot feature gate in the snapshot-controller
The snapshot-controller requires the CSIVolumeGroupSnapshot feature gate to be explicitly enabled to process VolumeGroupSnapshot objects. Run the following command to add the feature gate to the snapshot-controller deployment.
kubectl patch deployment snapshot-controller -n kube-system --type=json \
-p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--feature-gates=CSIVolumeGroupSnapshot=true"}]'❗ IMPORTANT: This command assumes that the
snapshot-controllerdeployment is in thekube-systemnamespace and that the first container in the pod spec is the snapshot-controller container, which is true for the standardexternal-snapshotterdeployment manifests. If your deployment differs, adjust the namespace and JSON patch path accordingly.
Enabling the CSIVolumeGroupSnapshot feature gate in the LINSTOR Operator
The LINSTOR Operator deploys a csi-snapshotter sidecar container inside the LINSTOR CSI controller pod. This sidecar handles the actual CSI CreateVolumeGroupSnapshot API calls to the LINSTOR CSI driver and also requires the feature gate to be enabled.
To enable it, patch the LinstorCluster resource to override the applicable environment variable in the csi-snapshotter container:
kubectl patch linstorcluster linstorcluster --type=merge --patch '
spec:
csiController:
podTemplate:
spec:
containers:
- name: csi-snapshotter
env:
- name: FEATURE_GATE_CSI_VOLUME_GROUP_SNAPSHOT
value: "true"
'After applying the patch, the LINSTOR Operator will restart the CSI controller pod with the updated environment variable. You can verify this by confirming the new pod picks up the setting.
kubectl get pods -n linbit-sds -l "app.kubernetes.io/component=linstor-csi-controller" \
-o jsonpath='{range .items[0].spec.containers[?(@.name=="csi-snapshotter")]}{range .env[*]}{.name}: {.value}\n{end}{end}'The output should show FEATURE_GATE_CSI_VOLUME_GROUP_SNAPSHOT: true.
Creating a volume group snapshot class
A VolumeGroupSnapshotClass defines how group snapshots should be created, in the same way that a VolumeSnapshotClass defines how individual volume snapshots should be created. Create a VolumeGroupSnapshotClass that references the LINSTOR CSI driver.
apiVersion: groupsnapshot.storage.k8s.io/v1
kind: VolumeGroupSnapshotClass
metadata:
name: linstor-group-snapshots
driver: linstor.csi.linbit.com
deletionPolicy: DeleteApply this to your cluster:
kubectl apply -f linstor-group-snapshots-class.yamlTaking a volume group snapshot
The following example demonstrates taking a crash-consistent group snapshot of a web application that uses two LINSTOR-backed PVCs: one for database storage, and one for user-uploaded media.
Grouping the PVCs by using a label
Volume group snapshots use Kubernetes label selectors to identify which PVCs should be included in a group. The two PVCs in this example were created with a group: my-app label. If your existing PVCs do not yet have this label, you can add it using the kubectl label command.
kubectl label pvc my-app-database group=my-app
kubectl label pvc my-app-uploads group=my-appThe following YAML creates the two PVCs used in this example:
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: my-app-database
namespace: default
labels:
group: my-app
spec:
storageClassName: linstor-csi-lvm-thin-r1
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: my-app-uploads
namespace: default
labels:
group: my-app
spec:
storageClassName: linstor-csi-lvm-thin-r1
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20GiCreating a volume group snapshot
With the PVCs labeled, create a VolumeGroupSnapshot that uses a label selector to identify all PVCs belonging to the group.
apiVersion: groupsnapshot.storage.k8s.io/v1
kind: VolumeGroupSnapshot
metadata:
name: my-app-snapshot
namespace: default
spec:
volumeGroupSnapshotClassName: linstor-group-snapshots
source:
selector:
matchLabels:
group: my-appApply this to your cluster:
kubectl apply -f my-app-snapshot.yamlVerifying the group snapshot
After creating the VolumeGroupSnapshot, you can watch its status until it reports READYTOUSE: true:
kubectl get volumegroupsnapshot my-app-snapshot --watchThe output will update as the snapshot progresses:
NAME READYTOUSE VOLUMEGROUPSNAPSHOTCLASS VOLUMEGROUPSNAPSHOTCONTENT CREATIONTIME AGE
my-app-snapshot linstor-group-snapshots 5s
my-app-snapshot false linstor-group-snapshots groupsnapcontent-81349afd-... 8s
my-app-snapshot true linstor-group-snapshots groupsnapcontent-81349afd-... 2s 12sWhen the VolumeGroupSnapshot becomes ready, the snapshot-controller automatically creates an individual VolumeSnapshot object for each PVC that was part of the group. You can list these individual snapshots, along with which PVC each was created from, by running.
kubectl get volumesnapshot -n defaultThe SOURCEPVC column identifies which PVC each individual VolumeSnapshot corresponds to:
NAME READYTOUSE SOURCEPVC RESTORESIZE CREATIONTIME
snapshot-7df4c9b9345f004500ba96484bf9a6f92d36b64df3607cc9e3164241fa6a70e8 true my-app-database 10Gi 2s
snapshot-ffbe7dedd160590f8101fa0950a12c4a39582a60aba57b8e6c23f55c402237ea true my-app-uploads 20Gi 2sRestoring from a group snapshot
To restore volumes from a group snapshot, create new PVCs that reference the individual VolumeSnapshot objects created by the group snapshot. The dataSource field on each PVC references the specific VolumeSnapshot for the volume you want to restore.
Use the kubectl get volumesnapshot output from the previous step to match the snapshot names to the PVCs you want to restore.
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: my-app-database-restore
namespace: default
spec:
storageClassName: linstor-csi-lvm-thin-r1
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
dataSource:
name: snapshot-7df4c9b9345f004500ba96484bf9a6f92d36b64df3607cc9e3164241fa6a70e8
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: my-app-uploads-restore
namespace: default
spec:
storageClassName: linstor-csi-lvm-thin-r1
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
dataSource:
name: snapshot-ffbe7dedd160590f8101fa0950a12c4a39582a60aba57b8e6c23f55c402237ea
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.ioAfter applying this YAML, both restore PVCs will be provisioned from the snapshot data and will reach Bound status quickly:
kubectl get pvc my-app-database-restore my-app-uploads-restoreNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS
my-app-database-restore Bound pvc-9293ffe4-84f1-4051-ad5c-81235746dd67 10Gi RWO linstor-csi-lvm-thin-r1
my-app-uploads-restore Bound pvc-46ee2702-ab4a-40f2-b502-c84ba1c80616 20Gi RWO linstor-csi-lvm-thin-r1You can then attach these restored PVCs to new pods or update your application deployment to point to the restored volumes.
Use cases for volume group snapshots
The following sections give overviews of some common use cases for volume group snapshots in Kubernetes.
Databases with separate data and log volumes
Many database engines, such as PostgreSQL or MySQL, perform better when data files and write-ahead logs (WALs) are on separate volumes. Volume group snapshots ensure that both volumes are captured at an identical point in time, which is essential for a consistent restore. A WAL snapshot taken even milliseconds after the data snapshot could reflect transactions not yet committed to the data volume.
Multi-tiered application backups
A web application might store relational data in a database volume and user-uploaded files in a separate media volume. A group snapshot captures both volumes in the same atomic operation. This guarantees that the database state and the uploaded file inventory are consistent with each other.
Taking a backup before upgrading
Before a major application upgrade that involves schema changes or data migrations across multiple PVCs, a volume group snapshot creates a single consistent rollback point. If the upgrade fails partway through, you can restore all volumes together to the state before the upgrade, without the risk of having partial migration artifacts.
Final thoughts
Using LINSTOR to take and restore volume group snapshots is a straightforward way to have crash-consistent multi-volume backups in Kubernetes, without requiring application-level quiescence or coordination. After enabling the feature gates, and a VolumeGroupSnapshotClass is in place, the workflow is as simple as labeling your PVCs and creating a VolumeGroupSnapshot object. This makes for a declarative, Kubernetes-native approach to multi-volume disaster recovery.
If you are new to LINSTOR in Kubernetes and want to try it using the convenience of container images in the official LINBIT container registry, drbd.io, reach out to the LINBIT team to request evaluation access. Alternatively, you can use the freely available upstream project, the Piraeus Operator, to deploy LINSTOR in Kubernetes.
