Authors: Joel Zhou and Michael Troutman
With a Kubernetes cluster backed by LINSTOR® managed persistent storage, you have a high-availability solution for applications and services running within your cluster. To complement this setup, you will likely want to consider a backup solution for your deployments. This is important for a variety of reasons but mitigating the effects of a local node disaster or having a potential migration tool are two valuable uses.
This tutorial will describe how to use Velero to back up a Kubernetes cluster namespace deployment. With LINBIT® SDS deployed into your cluster, LINSTOR automatically manages storage volume snapshots of your application’s persistently stored data.
Important: The backup solution described here is a hybrid solution. Your Kubernetes namespace deployment metadata will be backed up to an AWS S3 bucket, while your Kubernetes persistent storage data will be snapshotted and replicated locally using LINSTOR backed by DRBD. To achieve a full data backup, with metadata and persistent storage snapshots, to AWS S3, additional configuration is needed. Should you need this solution, you can contact LINBIT or stay tuned for an upcoming tutorial with these instructions.
To achieve this backup solution, you will need:
- A running Kubernetes cluster. For compatibility with the Kubernetes CSI Snapshotter, version 5.0.1, Kubernetes version >= 1.20 is recommended. Check the CSI Snapshotter’s releases page for recommendations for compatibility requirements for the version you are using.
- The latest version of LINBIT SDS installed into your Kubernetes cluster. If you have not installed LINBIT SDS yet, you can follow this quick start guide from LINBIT.
- A storage back end that supports a “snapshot” feature, for example, an LVM thin pool or ZFS pool.
- An enabled snapshot feature that is supported by the Kubernetes CSI driver, and that you have the snapshot-controller and snapshot-validation-webhook pods running in your Kubernetes cluster.
You can verify this using the following command:
$ kubectl get crds volumesnapshots.snapshot.storage.k8s.io \ volumesnapshotclasses.snapshot.storage.k8s.io \ volumesnapshotcontents.snapshot.storage.k8s.io
If you do not get any output from this command showing that these CRDs were already created, you will need to install the snapshot validation webhook and the volume snapshot controller into your Kubernetes cluster, using Helm charts. Installing the volume snapshot controller from the Helm chart will also install the necessary CRDs. You can verify this by repeating the
kubectl get crds command above, after running the following Helm commands:
$ helm repo add piraeus-charts https://piraeus.io/helm-charts/ $ helm install --namespace kube-system snapshot-validation-webhook \ piraeus-charts/snapshot-validation-webhook $ helm install --namespace kube-system snapshot-controller \ piraeus-charts/snapshot-controller
Note: If Helm is not installed, follow these installation instructions from the Helm project, or follow the instructions within the Integrating LINBIT SDS with Kubernetes quick start guide.
- Accessible Velero supported remote object storage. This tutorial uses an Amazon AWS S3 bucket. Other cloud provider storage solutions can also be used. See the Velero project’s AWS S3 compatible storage providers list for more information.
First, you will need to create a new storage class within the LINSTOR CSI provisioner. Create the following YAML configuration file that will describe the storage class.
$ cat > /tmp/01_storage_class.yaml <<EOF apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: linbit provisioner: linstor.csi.linbit.com parameters: autoPlace: "2" storagePool: MyThinSP resourceGroup: MyThinRG EOF
Important: The storagepool key value pair should reference an existing LINSTOR storage pool. Should you need to verify what storage pools are available, you can enter the command:
$ kubectl exec deployment/linstor-op-cs-controller -- linstor storage-pool list
Next, apply this configuration to your Kubernetes cluster.
$ kubectl apply -f /tmp/01_storage_class.yaml storageclass.storage.k8s.io/linbit created
Verify the storage class with this command:
$ kubectl get storageclasses NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE linbit linstor.csi.linbit.com Delete Immediate false 3s
Creating a Volume Snapshot Class for Velero
Next, create a YAML configuration for a volume snapshot class for Velero to use, by entering the following command:
$ cat > /tmp/02_snapshot_class.yaml <<EOF apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: linstor-csi-snapshot-class-s3 labels: velero.io/csi-volumesnapshot-class: "true" driver: linstor.csi.linbit.com deletionPolicy: Retain EOF
Next, create the volume snapshot class by applying your configuration file to your Kubernetes cluster.
$ kubectl apply -f /tmp/02_snapshot_class.yaml volumesnapshotclass.snapshot.storage.k8s.io/linstor-csi-snapshot-class-s3 created
Verify the volume snapshot class with this command:
$ kubectl get volumesnapshotclasses NAME DRIVER DELETIONPOLICY AGE linstor-csi-snapshot-class-s3 linstor.csi.linbit.com Retain 43s
Installing and Configuring Velero
This tutorial provides instructions for using Velero to back up Kubernetes metadata to an AWS S3 bucket, although according to Velero documentation, the plug-in may be compatible with other AWS S3 compatible storage providers. Before installing Velero, you will need an AWS S3 bucket available for Velero’s backup and restore sets. This tutorial uses a bucket called “linbit-backup”.
Next, create a secret file that has an AWS access key id and an AWS secret access key pair for a user with proper credentials for your S3 bucket. Use a text editor to create the file
/tmp/aws.key with the following contents:
[default] aws_access_key_id=<your_access_key_id> aws_secret_access_key=<your_secret_access_key>
Determine the appropriate version of Velero for your version of Kubernetes, then download the Velero tar archive from the official GitHub repository. This article uses Kubernetes version 1.23 and Velero version 1.8.1.
$ vel_vers=v1.8.1 # Download Velero $ curl -fsSL -o velero-$vel_vers-linux-amd64.tar.gz \ https://github.com/vmware-tanzu/velero/releases/download/$vel_vers/velero-$vel_vers-linux-amd64.tar.gz
After verifying the checksum of the downloaded tar archive, extract it, and copy the extracted
velero executable file into a directory in your
Installing Velero Into Kubernetes
Next, create environment variables for your AWS S3 bucket name and your AWS region, as well as variables for the version numbers of the Velero plug-in for AWS and plug-in for the CSI driver that are compatible with your version of Velero. This article uses versions 1.4.1 and 0.2.0 respectively, for Velero version 1.8.1. Set variables accordingly, for example:
$ aws_bucket=linbit-velero-backup $ aws_region=us-east-2 $ aws_plugin_vers=v1.4.1 $ csi_plugin_vers=v0.2.0
Then, install Velero.
$ velero install --features=EnableCSI --use-volume-snapshots \ --plugins=velero/velero-plugin-for-aws:$aws_plugin_vers,velero/velero-plugin-for-csi:$csi_plugin_vers \ --provider aws --bucket $aws_bucket --secret-file /tmp/aws.key \ --backup-location-config region=$aws_region --snapshot-location-config region=$aws_region
You should get output about Velero resources being created and eventually get confirmation of a successful deployment and installation.
[...] Deployment/velero: created Velero is installed! Use 'kubectl logs deployment/velero -n velero' to view the status.
After a while, entering a
kubectl get pods -A command should show that a new Velero server pod, named “velero”, is up and running. You can also monitor the status of the Velero deployment with the following command.
$ watch kubectl logs deployment/velero -n velero
Configuring Velero and Installing an Example Pod
With Velero installed into your Kubernetes cluster, you can configure it and test its backup and restore capabilities by creating an example pod.
Enabling the CSI Feature on the Velero Client Configuration
To prepare to backup your Kubernetes metadata using Velero, you enabled the Velero CSI feature on the Velero “server” pod, by using the
--features flag in your
velero install command. You also need to enable this feature within the Velero client, by entering the following command.
$ velero client config set features=EnableCSI
You can verify that the feature flag is set by entering the following command.
$ velero client config get features features: EnableCSI
Creating an Example Pod
In this article, you will create a basic MySQL pod to test Velero’s backup and restore capabilities.
Changing the Kubernetes Namespace
First, create a new Kubernetes namespace for testing Velero.
$ kubectl create namespace velero-testing
Next, change your context to the “velero-testing” namespace, so that the
kubectl commands that follow will happen within the “velero-testing” namespace.
$ kubectl config set-context --current --namespace velero-testing Context "kubernetes-admin@kubernetes" modified.
Creating a Persistent Volume Claim
To have LINSTOR manage your example pod’s persistent storage, you will need to create a persistent volume claim (PVC) that references the “linbit” storage class you created earlier. You can do this by first creating a YAML configuration file for the persistent volume claim.
$ cat > /tmp/03_pvc.yaml <<EOF kind: PersistentVolumeClaim apiVersion: v1 metadata: name: mysql-pvc namespace: velero-testing spec: accessModes: - ReadWriteOnce resources: requests: storage: 4Gi storageClassName: linbit EOF
Next, apply the configuration to your Kubernetes cluster.
$ kubectl apply -f /tmp/03_pvc.yaml persistentvolumeclaim/mysql-pvc created
You can verify the deployment of your PVC by using a
kubectl get command.
$ kubectl get persistentvolumeclaims NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mysql-pvc Bound pvc-[...] 4Gi RWO linbit 22s
Creating an Example Pod
With a PVC created, you can create an example MySQL pod that has its data directory (
/var/lib/mysql) on the PVC. First, create the pod’s YAML configuration file.
$ cat > /tmp/04_mysql.yaml <<EOF apiVersion: apps/v1 kind: Deployment metadata: name: mysql namespace: velero-testing spec: replicas: 1 selector: matchLabels: app: mysql strategy: type: Recreate template: metadata: labels: app: mysql spec: containers: - image: mysql:8.0 name: mysql env: - name: MYSQL_ROOT_PASSWORD value: linbit ports: - containerPort: 3306 name: mysql volumeMounts: - name: mysql-persistent-storage mountPath: /var/lib/mysql volumes: - name: mysql-persistent-storage persistentVolumeClaim: claimName: mysql-pvc EOF
Next, apply the configuration to your Kubernetes cluster.
$ kubectl apply -f /tmp/04_mysql.yaml deployment.apps/mysql created
After some time, your example pod should come up within the “velero-testing” namespace. You can confirm it is up and running using a basic
kubectl get pods command.
$ kubectl get pods NAME READY STATUS RESTARTS AGE mysql-[...] 1/1 Running 0 121m
You can also watch your pod come up in real time, using the following command:
$ watch kubectl get pods
Testing Velero’s Backup and Restore Functions
With Velero installed and configured and an example pod up and running, you can next test Velero’s backup capabilities to an AWS S3 bucket.
Create a Unique Reference File Within Your MySQL Pod
For testing Velero’s backup and restore capabilities, first create a unique reference file in your pod’s MySQL data directory. You can check for this reference file after restoring your backup. This will confirm that LINSTOR created and restored from a snapshot of your application’s persistent storage volume.
Borrowing an example from the Velero documentation, log into your MySQL example pod and create a unique file, then get its checksum for verification purposes later.
$ kubectl exec -it deployment/mysql -- /bin/bash root@mysql-[...]:/# while true; do echo -n "LINBIT " >> /var/lib/mysql/velero.testing.file; done ^C root@mysql-[...]:/# cksum /var/lib/mysql/velero.testing.file 2341652068 1708749 /var/lib/mysql/velero.testing.file root@mysql-[...]:/# exit
Backing Up Your Application’s Deployment
Create an AWS S3 stored backup of your application’s Kubernetes deployment metadata by using the following Velero command. Through the CSI driver, because you configured LINSTOR to back the storage class underlying your application’s PVC, LINSTOR will automatically take a volume snaphot as part of the Velero backup.
$ velero backup create backup01 --include-namespaces velero-testing --wait Backup request "backup01" submitted successfully. Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background. ................... Backup completed with status: Completed. You may check for more information using the commands `velero backup describe backup01` and `velero backup logs backup01`.
After some time, the backup will complete and you should find a “backups/backup01” folder withing your AWS S3 bucket and within that folder, archives of your deployment’s metadata.
Next, verify that your Velero backup includes a volume snapshot.
$ velero describe backups backup01 --details Name: backup01 [...] CSI Volume Snapshots: Snapshot Content Name: snapcontent-77401f10-091a-484a-adf7-56dd58844e48 Storage Snapshot ID: snapshot-77401f10-091a-484a-adf7-56dd58844e48 Snapshot Size (bytes): 10737418240 Ready to use: true
To further verify the snapshot, you can SSH into one of your diskful Kubernetes worker nodes. You can run the command
lsblk to show that there is an LVM volume with “snapshot” in its name and that it has a UID that matches the CSI volume snapshot ID listed using the
velero describe backups command that you ran earlier.
Within Kubernetes, you can also verify that creating a Velero backup also created a snapshot referenced in Kubernetes’ VolumeSnapshotContent, by using the
kubectl get VolumeSnapshotContent -A command.
Testing Your Backup
Before disaster or migration needs require it, test that you can restore your application’s deployment backup to your Kubernetes cluster.
Deleting Unique Test File and Application’s Namespace
First, enter your MySQL example pod, delete the unique test file you created earlier, and create a new empty file that will serve as a second test that your backup was restored.
$ kubectl exec -it deployment/mysql -- /bin/bash root@mysql-[...]:/# rm /var/lib/mysql/velero.testing.file root@mysql-[...]:/# touch /var/lib/mysql/velero.restore.test.file root@mysql-[...]:/# ls /var/lib/mysql | grep velero velero.restore.test.file root@mysql-[...]:/# exit
Next, delete your application’s Kubernetes namespace.
$ kubectl delete namespace velero-testing namespace "velero-testing" deleted
Verify that your MySQL example pod is gone.
$ kubectl get pods -A | awk 'NR == 1 || /mysql/' NAMESPACE NAME READY STATUS RESTARTS AGE
Restoring Your Backup
Next, use Velero to restore your backup. LINSTOR will automatically restore the storage class volume from the LVM volume snapshot that was created as part of your backup.
$ velero restore create restore01 --from-backup backup01 --wait Restore request "restore01" submitted successfully. Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background. ............ Restore completed with status: Completed. You may check for more information using the commands `velero restore describe restore01` and `velero restore logs restore01`.
Verifying Your Restore
After restoring your Velero backup, confirm that your MySQL example pod has returned.
$ kubectl get pods -A | awk 'NR == 1 || /mysql/' NAMESPACE NAME READY STATUS RESTARTS AGE velero-testing mysql-[...] 1/1 Running 0 96s
Enter your application’s pod and verify that LINSTOR restored the storage volume from the LVM snapshot it created as part of the Velero backup.
$ kubectl exec -it deployment/mysql -- /bin/bash root@mysql-[...]:/# ls /var/lib/mysql | grep velero velero.testing.file root@mysql-[...]:/# cksum /var/lib/mysql/velero.testing.file 2341652068 1708749 /var/lib/mysql/velero.testing.file root@mysql-[...]:/# exit
The above commands show that the MySQL data directory has your original test file and you confirmed the file’s contents by using a checksum command. The command output also shows that the second test file you created, “velero.restore.test.file”, is gone because you created it after you made your backup. This further confirms your successful backup restore.
Using Velero, LINBIT SDS, and snapshot capable storage volumes, you have created a solid system for backing up and restoring your Kubernetes deployments. LINBIT invites you to further test the use cases of this solution and to contact us with questions specific to your environment, deployment, and application needs.
In this tutorial, your Kubernetes application deployment’s metadata was backed up to AWS S3 storage. However, your LINSTOR managed persistent data storage was still local, although it was redundant across multiple worker nodes. If you want an off-site disaster recovery solution for your deployment’s persistent data storage, you will also need to configure snapshots and backup shipping. Stay tuned for an upcoming tutorial with these instructions. Should you need a solution specific to your deployment, you can contact the experts at LINBIT.