Backing Up Your Kubernetes Deployments With Velero & LINSTOR

Authors: Joel Zhou and Michael Troutman

Introduction

With a Kubernetes cluster backed by LINSTOR® managed persistent storage, you have a high-availability solution for applications and services running within your cluster. To complement this setup, you will likely want to consider a backup solution for your deployments. This is important for a variety of reasons but mitigating the effects of a local node disaster or having a potential migration tool are two valuable uses.

This tutorial will describe how to use Velero to back up a Kubernetes cluster namespace deployment. With LINBIT® SDS deployed into your cluster, LINSTOR automatically manages storage volume snapshots of your application’s persistently stored data.

Important: The backup solution described here is a hybrid solution. Your Kubernetes namespace deployment metadata will be backed up to an AWS S3 bucket, while your Kubernetes persistent storage data will be snapshotted and replicated locally using LINSTOR backed by DRBD. To achieve a full data backup, with metadata and persistent storage snapshots, to AWS S3, additional configuration is needed. Should you need this solution, you can contact LINBIT or stay tuned for an upcoming tutorial with these instructions.

Prerequisites

To achieve this backup solution, you will need:

  • A running Kubernetes cluster. For compatibility with the Kubernetes CSI Snapshotter, version 5.0.1, Kubernetes version >= 1.20 is recommended. Check the CSI Snapshotter’s releases page for recommendations for compatibility requirements for the version you are using.
  • The latest version of LINBIT SDS installed into your Kubernetes cluster. If you have not installed LINBIT SDS yet, you can follow this quick start guide from LINBIT.
  • A storage back end that supports a “snapshot” feature, for example, an LVM thin pool or ZFS pool.
  • An enabled snapshot feature that is supported by the Kubernetes CSI driver, and that you have the snapshot-controller and snapshot-validation-webhook pods running in your Kubernetes cluster.

You can verify this using the following command:

$ kubectl get crds volumesnapshots.snapshot.storage.k8s.io \
volumesnapshotclasses.snapshot.storage.k8s.io \
volumesnapshotcontents.snapshot.storage.k8s.io

If you do not get any output from this command showing that these CRDs were already created, you will need to install the snapshot validation webhook and the volume snapshot controller into your Kubernetes cluster, using Helm charts. Installing the volume snapshot controller from the Helm chart will also install the necessary CRDs. You can verify this by repeating the kubectl get crds command above, after running the following Helm commands:

$ helm repo add piraeus-charts https://piraeus.io/helm-charts/
$ helm install --namespace kube-system snapshot-validation-webhook \
piraeus-charts/snapshot-validation-webhook
$ helm install --namespace kube-system snapshot-controller \
piraeus-charts/snapshot-controller

Note: If Helm is not installed, follow these installation instructions from the Helm project, or follow the instructions within the Integrating LINBIT SDS with Kubernetes quick start guide.

  • Accessible Velero supported remote object storage. This tutorial uses an Amazon AWS S3 bucket. Other cloud provider storage solutions can also be used. See the Velero project’s AWS S3 compatible storage providers list for more information.

Getting started

First, you will need to create a new storage class within the LINSTOR CSI provisioner. Create the following YAML configuration file that will describe the storage class.

$ cat > /tmp/01_storage_class.yaml <<EOF 
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linbit
provisioner: linstor.csi.linbit.com
parameters:
  autoPlace: "2"
  storagePool: MyThinSP
  resourceGroup: MyThinRG
EOF

Important: The storagepool key value pair should reference an existing LINSTOR storage pool. Should you need to verify what storage pools are available, you can enter the command:

$ kubectl exec deployment/linstor-op-cs-controller -- linstor storage-pool list

Next, apply this configuration to your Kubernetes cluster.

$ kubectl apply -f /tmp/01_storage_class.yaml
storageclass.storage.k8s.io/linbit created

Verify the storage class with this command:

$ kubectl get storageclasses
NAME     PROVISIONER              RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
linbit   linstor.csi.linbit.com   Delete          Immediate           false                  3s

Creating a Volume Snapshot Class for Velero

Next, create a YAML configuration for a volume snapshot class for Velero to use, by entering the following command:

$ cat > /tmp/02_snapshot_class.yaml <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: linstor-csi-snapshot-class-s3
  labels:
    velero.io/csi-volumesnapshot-class: "true"
driver: linstor.csi.linbit.com
deletionPolicy: Retain
EOF

Next, create the volume snapshot class by applying your configuration file to your Kubernetes cluster.

$ kubectl apply -f /tmp/02_snapshot_class.yaml
volumesnapshotclass.snapshot.storage.k8s.io/linstor-csi-snapshot-class-s3 created

Verify the volume snapshot class with this command:

$ kubectl get volumesnapshotclasses
NAME                            DRIVER                   DELETIONPOLICY   AGE
linstor-csi-snapshot-class-s3   linstor.csi.linbit.com   Retain           43s

Installing and Configuring Velero

This tutorial provides instructions for using Velero to back up Kubernetes metadata to an AWS S3 bucket, although according to Velero documentation, the plug-in may be compatible with other AWS S3 compatible storage providers. Before installing Velero, you will need an AWS S3 bucket available for Velero’s backup and restore sets. This tutorial uses a bucket called “linbit-backup”.

Next, create a secret file that has an AWS access key id and an AWS secret access key pair for a user with proper credentials for your S3 bucket. Use a text editor to create the file /tmp/aws.key with the following contents:

[default]
aws_access_key_id=<your_access_key_id>
aws_secret_access_key=<your_secret_access_key>

Downloading Velero

Determine the appropriate version of Velero for your version of Kubernetes, then download the Velero tar archive from the official GitHub repository. This article uses Kubernetes version 1.23 and Velero version 1.8.1.

$ vel_vers=v1.8.1
# Download Velero
$ curl -fsSL -o velero-$vel_vers-linux-amd64.tar.gz \ https://github.com/vmware-tanzu/velero/releases/download/$vel_vers/velero-$vel_vers-linux-amd64.tar.gz

After verifying the checksum of the downloaded tar archive, extract it, and copy the extracted velero executable file into a directory in your $PATH.

Installing Velero Into Kubernetes

Next, create environment variables for your AWS S3 bucket name and your AWS region, as well as variables for the version numbers of the Velero plug-in for AWS and plug-in for the CSI driver that are compatible with your version of Velero. This article uses versions 1.4.1 and 0.2.0 respectively, for Velero version 1.8.1. Set variables accordingly, for example:

$ aws_bucket=linbit-velero-backup
$ aws_region=us-east-2
$ aws_plugin_vers=v1.4.1
$ csi_plugin_vers=v0.2.0

Then, install Velero.

$ velero install --features=EnableCSI --use-volume-snapshots \
--plugins=velero/velero-plugin-for-aws:$aws_plugin_vers,velero/velero-plugin-for-csi:$csi_plugin_vers \
--provider aws --bucket $aws_bucket --secret-file /tmp/aws.key \
--backup-location-config region=$aws_region --snapshot-location-config region=$aws_region

You should get output about Velero resources being created and eventually get confirmation of a successful deployment and installation.

[...]
Deployment/velero: created
Velero is installed! Use 'kubectl logs deployment/velero -n velero' to view the status.

After a while, entering a kubectl get pods -A command should show that a new Velero server pod, named “velero”, is up and running. You can also monitor the status of the Velero deployment with the following command.

$ watch kubectl logs deployment/velero -n velero

Configuring Velero and Installing an Example Pod

With Velero installed into your Kubernetes cluster, you can configure it and test its backup and restore capabilities by creating an example pod.

Enabling the CSI Feature on the Velero Client Configuration

To prepare to backup your Kubernetes metadata using Velero, you enabled the Velero CSI feature on the Velero “server” pod, by using the --features flag in your velero install command. You also need to enable this feature within the Velero client, by entering the following command.

$ velero client config set features=EnableCSI

You can verify that the feature flag is set by entering the following command.

$ velero client config get features
features: EnableCSI

Creating an Example Pod

In this article, you will create a basic MySQL pod to test Velero’s backup and restore capabilities.

Changing the Kubernetes Namespace

First, create a new Kubernetes namespace for testing Velero.

$ kubectl create namespace velero-testing

Next, change your context to the “velero-testing” namespace, so that the kubectl commands that follow will happen within the “velero-testing” namespace.

$ kubectl config set-context --current --namespace velero-testing
Context "kubernetes-admin@kubernetes" modified.

Creating a Persistent Volume Claim

To have LINSTOR manage your example pod’s persistent storage, you will need to create a persistent volume claim (PVC) that references the “linbit” storage class you created earlier. You can do this by first creating a YAML configuration file for the persistent volume claim.

$ cat > /tmp/03_pvc.yaml <<EOF
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysql-pvc
  namespace: velero-testing
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 4Gi
  storageClassName: linbit
EOF

Next, apply the configuration to your Kubernetes cluster.

$ kubectl apply -f /tmp/03_pvc.yaml
persistentvolumeclaim/mysql-pvc created

You can verify the deployment of your PVC by using a kubectl get command.

$ kubectl get persistentvolumeclaims 
NAME        STATUS   VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mysql-pvc   Bound    pvc-[...]   4Gi        RWO            linbit         22s

Creating an Example Pod

With a PVC created, you can create an example MySQL pod that has its data directory (/var/lib/mysql) on the PVC. First, create the pod’s YAML configuration file.

$ cat > /tmp/04_mysql.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  namespace: velero-testing
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - image: mysql:8.0
          name: mysql
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: linbit
          ports:
            - containerPort: 3306
              name: mysql
          volumeMounts:
            - name: mysql-persistent-storage
              mountPath: /var/lib/mysql
      volumes:
        - name: mysql-persistent-storage
          persistentVolumeClaim:
            claimName: mysql-pvc
EOF

Next, apply the configuration to your Kubernetes cluster.

$ kubectl apply -f /tmp/04_mysql.yaml
deployment.apps/mysql created

After some time, your example pod should come up within the “velero-testing” namespace. You can confirm it is up and running using a basic kubectl get pods command.

$ kubectl get pods
NAME					READY   STATUS    RESTARTS   AGE
mysql-[...]				1/1     Running   0          121m

You can also watch your pod come up in real time, using the following command:

$ watch kubectl get pods

Testing Velero’s Backup and Restore Functions

With Velero installed and configured and an example pod up and running, you can next test Velero’s backup capabilities to an AWS S3 bucket.

Create a Unique Reference File Within Your MySQL Pod

For testing Velero’s backup and restore capabilities, first create a unique reference file in your pod’s MySQL data directory. You can check for this reference file after restoring your backup. This will confirm that LINSTOR created and restored from a snapshot of your application’s persistent storage volume.

Borrowing an example from the Velero documentation, log into your MySQL example pod and create a unique file, then get its checksum for verification purposes later.

$ kubectl exec -it deployment/mysql -- /bin/bash
root@mysql-[...]:/# while true; do echo -n "LINBIT " >> /var/lib/mysql/velero.testing.file; done
^C
root@mysql-[...]:/# cksum /var/lib/mysql/velero.testing.file
2341652068 1708749 /var/lib/mysql/velero.testing.file
root@mysql-[...]:/# exit

Backing Up Your Application’s Deployment

Create an AWS S3 stored backup of your application’s Kubernetes deployment metadata by using the following Velero command. Through the CSI driver, because you configured LINSTOR to back the storage class underlying your application’s PVC, LINSTOR will automatically take a volume snaphot as part of the Velero backup.

$ velero backup create backup01 --include-namespaces velero-testing --wait
Backup request "backup01" submitted successfully.
Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background.
...................
Backup completed with status: Completed. You may check for more information using the commands `velero backup describe backup01` and `velero backup logs backup01`.

After some time, the backup will complete and you should find a “backups/backup01” folder withing your AWS S3 bucket and within that folder, archives of your deployment’s metadata.

Next, verify that your Velero backup includes a volume snapshot.

$ velero describe backups backup01 --details
Name:         backup01
[...]
CSI Volume Snapshots:
Snapshot Content Name: snapcontent-77401f10-091a-484a-adf7-56dd58844e48
  Storage Snapshot ID: snapshot-77401f10-091a-484a-adf7-56dd58844e48
  Snapshot Size (bytes): 10737418240
  Ready to use: true

To further verify the snapshot, you can SSH into one of your diskful Kubernetes worker nodes. You can run the command lsblk to show that there is an LVM volume with “snapshot” in its name and that it has a UID that matches the CSI volume snapshot ID listed using the velero describe backups command that you ran earlier.

Within Kubernetes, you can also verify that creating a Velero backup also created a snapshot referenced in Kubernetes’ VolumeSnapshotContent, by using the kubectl get VolumeSnapshotContent -A command.

Testing Your Backup

Before disaster or migration needs require it, test that you can restore your application’s deployment backup to your Kubernetes cluster.

Deleting Unique Test File and Application’s Namespace

First, enter your MySQL example pod, delete the unique test file you created earlier, and create a new empty file that will serve as a second test that your backup was restored.

$ kubectl exec -it deployment/mysql -- /bin/bash
root@mysql-[...]:/# rm /var/lib/mysql/velero.testing.file
root@mysql-[...]:/# touch /var/lib/mysql/velero.restore.test.file
root@mysql-[...]:/# ls /var/lib/mysql | grep velero
velero.restore.test.file
root@mysql-[...]:/# exit

Next, delete your application’s Kubernetes namespace.

$ kubectl delete namespace velero-testing
namespace "velero-testing" deleted

Verify that your MySQL example pod is gone.

$ kubectl get pods -A | awk 'NR == 1 || /mysql/'
NAMESPACE			NAME					READY   STATUS    RESTARTS   AGE

Restoring Your Backup

Next, use Velero to restore your backup. LINSTOR will automatically restore the storage class volume from the LVM volume snapshot that was created as part of your backup.

$ velero restore create restore01 --from-backup backup01 --wait
Restore request "restore01" submitted successfully.
Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background.
............
Restore completed with status: Completed. You may check for more information using the commands `velero restore describe restore01` and `velero restore logs restore01`.

Verifying Your Restore

After restoring your Velero backup, confirm that your MySQL example pod has returned.

$ kubectl get pods -A | awk 'NR == 1 || /mysql/'
NAMESPACE			NAME					READY   STATUS    RESTARTS   AGE
velero-testing		mysql-[...]				1/1     Running   0          96s

Enter your application’s pod and verify that LINSTOR restored the storage volume from the LVM snapshot it created as part of the Velero backup.

$ kubectl exec -it deployment/mysql -- /bin/bash
root@mysql-[...]:/# ls /var/lib/mysql | grep velero
velero.testing.file
root@mysql-[...]:/# cksum /var/lib/mysql/velero.testing.file
2341652068 1708749 /var/lib/mysql/velero.testing.file
root@mysql-[...]:/# exit

The above commands show that the MySQL data directory has your original test file and you confirmed the file’s contents by using a checksum command. The command output also shows that the second test file you created, “velero.restore.test.file”, is gone because you created it after you made your backup. This further confirms your successful backup restore.

Conclusion

Using Velero, LINBIT SDS, and snapshot capable storage volumes, you have created a solid system for backing up and restoring your Kubernetes deployments. LINBIT invites you to further test the use cases of this solution and to contact us with questions specific to your environment, deployment, and application needs.

In this tutorial, your Kubernetes application deployment’s metadata was backed up to AWS S3 storage. However, your LINSTOR managed persistent data storage was still local, although it was redundant across multiple worker nodes. If you want an off-site disaster recovery solution for your deployment’s persistent data storage, you will also need to configure snapshots and backup shipping. Stay tuned for an upcoming tutorial with these instructions. Should you need a solution specific to your deployment, you can contact the experts at LINBIT.

Share on facebook
Share on twitter
Share on linkedin
Share on reddit
Share on whatsapp
Share on vk
Share on email

Share this post

Joel Zhou

Joel Zhou

Joel Zhou is a Solutions Architect at LINBIT, and has more than 15 years of experience in open source software, especially focusing on enterprise infrastructure (cloud computing, software-defined storage, etc.). He's been to the basecamp of Everest, dived in the ocean in the Maldives, and is a coffee lover and Rock & Roll fan.

Talk to us

LINBIT is committed to protecting and respecting your privacy, and we’ll only use your personal information to administer your account and to provide the products and services you requested from us. From time to time, we would like to contact you about our products and services, as well as other content that may be of interest to you. If you consent to us contacting you for this purpose, please tick above to say how you would like us to contact you.

You can unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.

By clicking submit below, you consent to allow LINBIT to store and process the personal information submitted above to provide you the content requested.

Talk to us

LINBIT is committed to protecting and respecting your privacy, and we’ll only use your personal information to administer your account and to provide the products and services you requested from us. From time to time, we would like to contact you about our products and services, as well as other content that may be of interest to you. If you consent to us contacting you for this purpose, please tick above to say how you would like us to contact you.

You can unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.

By clicking submit below, you consent to allow LINBIT to store and process the personal information submitted above to provide you the content requested.