Abstracting Persistent Storage Across Environments With LINBIT SDS

Kubernetes can be thought of as an infrastructure abstraction layer for on-premise and cloud computing. The applications you deploy into one cloud platform or on-premise Kubernetes cluster can usually be deployed into another cluster using the same manifests and automations. That is a huge win for developers and development life cycles in terms of simplicity and productivity.

This concept can break down a bit when it comes to stateful applications that require persistent storage, because storage solutions will often differ between environments. Moving storage between disparate environment is much more difficult than moving an application. Also, the underlying persistent storage solution in one cluster might offer a different feature set than what might be available in another cluster. For example, Akamai’s Linode Kubernetes Engine (LKE) only allows 8 persistent volumes to be attached to a single compute node while Amazon’s Elastic Kubernetes Service (EKS) is limited to 39. Another example, EKS’s CSI driver supports volume snapshots, while LKE’s does not. This means how users create and use their persistent volumes, or how your operations team manages those persistent volumes, will differ. This can increase the complexity and introduce “pet-like” characteristics in organizations who would rather treat their clusters like “cattle”.

LINSTOR® can help to better abstract away the differences in persistent storage layers between disparate environments by providing a single feature rich storage solution that works the same across all environments. This is true regardless of whether you’re running many on-premise Kubernetes clusters with varying storage back ends, or many clusters in multiple public clouds, or both. Along with unified features, using LINSTOR across various environments enables migration of persistent storage between them, which can simplify disaster recovery strategies, as well as enable cloud migrations or cloud repatriations. The latter is what I will demonstrate in this blog.

Prerequisites for Migrating Persistent Volumes Between Environments using LINSTOR

Obviously, you’re going to need two clusters, and they’ll need to have the LINSTOR Operator (or the Piraeus Operator for FLOSS users) deployed. This subject is covered in the linked documentation as well as many other LINBIT® blogs and knowledge base articles. I’ll refer to these clusters as “source” and “target” throughout this blog post, with the source cluster being where your stateful application is currently running, and the target cluster being the cluster you intend to migrate your stateful application to. In the configuration and command examples throughout this post Amazon’s EKS hosts the source cluster, and Akamai’s LKE hosts the target cluster – with the kubectl config defined with eks and lke as each cluster’s respective context.

In addition to the LINSTOR Operator, you’ll also need to have added snapshot support to your clusters by deploying LINSTOR’s snapshot controller. Volume snapshot support was moved to general availability (GA) in Kubernetes 1.20, so provided that you’re using Kubernetes v1.20 or better, adding snapshot support to your clusters is easily done using the Helm charts provided by the Piraeus project.

Lastly, you’ll need an S3 compatible bucket configured somewhere that both your source and target clusters can access. The examples in this blog will use an AWS S3 bucket, but you can use any S3 compatible object storage such as Akamai’s Object Storage, or Minio’s S3 compatible buckets. To configure S3 compatible object storage, you’ll need to know the name of the bucket, the S3 endpoint, as well as the signing region for SigV4 signing of requests if applicable to your object storage. You’ll also need the appropriate access key and secret access key needed for reading and writing to the S3 compatible object storage.

NOTE: Because I will be running commands against two different clusters, I will use kubectl config use-context before commands to better demonstrate which cluster is being configured.

Configuring the S3 Snapshot Class in the Source Cluster

LINSTOR can send and receive snapshots using what LINSTOR refers to as “remotes”. In the source Kubernetes cluster, you will need to define the LINSTOR remote using a VolumeSnapshotClass, as well as the Secret that LINSTOR will need to access the S3 remote. In the examples below, we’ll create a VolumeSnapshotClass named linstor-csi-snapshot-class-aws-s3-us-west-2, which will define a remote within LINSTOR named aws-s3-us-west-2, that references an S3 bucket named itzabucket123 in the us-west-2 signing region, which can be accessed using the s3.us-west-2.amazonaws.com S3 endpoint. We’ll also create a Secret that contains the credentials required to access the bucket.

$ cat << EOF > linstor-s3-snapclass.yaml
kind: VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1
metadata:
  name: linstor-csi-snapshot-class-aws-s3-us-west-2
driver: linstor.csi.linbit.com
deletionPolicy: Retain
parameters:
  snap.linstor.csi.linbit.com/type: S3
  snap.linstor.csi.linbit.com/remote-name: aws-s3-us-west-2
  snap.linstor.csi.linbit.com/allow-incremental: "false"
  snap.linstor.csi.linbit.com/s3-bucket: itzabucket123
  snap.linstor.csi.linbit.com/s3-endpoint: s3.us-west-2.amazonaws.com
  snap.linstor.csi.linbit.com/s3-signing-region: us-west-2
  snap.linstor.csi.linbit.com/s3-use-path-style: "false"
  # Refer here to the secret that holds access and secret key for the S3 endpoint.
  # See below for an example.
  csi.storage.k8s.io/snapshotter-secret-name: linstor-csi-aws-s3-us-west-2-access
  csi.storage.k8s.io/snapshotter-secret-namespace: default
---
kind: Secret
apiVersion: v1
metadata:
  name: linstor-csi-aws-s3-us-west-2-access
  namespace: default
immutable: true
type: linstor.csi.linbit.com/s3-credentials.v1
stringData:
  access-key: _REDACTED_AK_
  secret-key: _REDACTED_SK_
EOF

$ kubectl config use-context eks
$ kubectl apply -f linstor-s3-snapclass.yaml

NOTE: The VolumeSnapshotClass only needs to be defined on the source cluster where the snapshot will originate at this point. However, we will use the LINSTOR command line client on the target cluster, as well as this YAML manifest, to configure the LINSTOR remote in the target cluster before restoring the snapshot, so be sure you keep the VolumeSnapshotClass and Secret manifests handy.

Creating and Shipping the Snapshot in the Source Cluster

To create the snapshot and ship it to the S3 remote, pick any PersistentVolumeClaim in your cluster created from a LINSTOR StorageClass, and reference it as the source in the example YAML below:

$ cat << EOF > linstor-s3-snapshot.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: my-first-linstor-snapshot
spec:
  volumeSnapshotClassName: linstor-csi-snapshot-class-aws-s3-us-west-2
  source:
    persistentVolumeClaimName: important-data
EOF

$ kubectl config use-context eks
$ kubectl apply -f linstor-s3-snapshot.yaml

In a few moments, you should see that your VolumeSnapshot has become ready to use.

$ kubectl config use-context eks
$ kubectl get volumesnapshots \
    --no-headers -o custom-columns=":metadata.name,:status.readyToUse"

my-first-linstor-snapshot   true

Additionally, if you browse the S3 bucket using the AWS console you should see that your PVC data has been shipped to your S3 bucket. At this point, if you intend to completely move the application from your source to target cluster, you can stop the application in the source cluster.

Restoring the Snapshot into the Target Cluster

With the snapshot saved in S3, you can now migrate the persistent volume into your target cluster. To do so, we’ll need to first configure the LINSTOR remote in the LINSTOR controller of the target cluster. Replace the S3 bucket placeholder credentials in the example commands below with your own S3 bucket credentials. Also, be sure to switch your context to the target cluster.

$ kubectl config use-context lke
$ kubectl exec -it -n linstor deployments/linstor-op-cs-controller -- \
    linstor remote create s3 aws-s3-us-west-2 s3.us-west-2.amazonaws.com \
    itzabucket123 us-west-2 _REDACTED_AK_ _REDACTED_SK_

After configuring the remote, you should be able to list the contents of your LINSTOR remote and see the snapshot you created. Note the Snapshot name (which starts with snapshot-[...]) from the LINSTOR output, as this will be used in subsequent steps.

$ kubectl exec -it -n linstor deployments/linstor-op-cs-controller -- \
    linstor backup list aws-s3-us-west-2
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ Resource       ┊ Snapshot                                      ┊ Finished at         ┊ Based On ┊ Status  ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ pvc-7ba3b[...] ┊ snapshot-4372a910-87ee-460e-8f41-16131bc93f18 ┊ 2023-02-14 20:38:18 ┊          ┊ Success ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Using the same VolumeSnapshotClass and Secret manifests as used in the source cluster, create the VolumeSnapshotClass and Secret in the target cluster.

$ kubectl config use-context lke
$ kubectl apply -f linstor-s3-snapclass.yaml

Now, you can create the Snapshot and SnapshotContent. The spec.source.snapshotHandle defined in the VolumeSnapshotContent below refers to the Snapshot name from the linstor backup list aws-s3-us-west-2 output in a previous step, and the spec.volumeSnapshotClassName refers to the VolumeSnapshotClass created in a previous step.

$ cat << EOF > linstor-s3-snap-restore.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: important-data-restored-from-s3
  namespace: default
spec:
  source:
    volumeSnapshotContentName: important-data-restored-content-from-s3
  volumeSnapshotClassName: linstor-csi-snapshot-class-aws-s3-us-west-2
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotContent
metadata:
  name: important-data-restored-content-from-s3
spec:
  deletionPolicy: Delete
  driver: linstor.csi.linbit.com
  source:
    snapshotHandle: snapshot-4372a910-87ee-460e-8f41-16131bc93f18
  volumeSnapshotClassName: linstor-csi-snapshot-class-aws-s3-us-west-2
  volumeSnapshotRef:
    apiVersion: snapshot.storage.k8s.io/v1
    kind: VolumeSnapshot
    name: important-data-restored-from-s3
    namespace: default
EOF

$ kubectl config use-context lke
$ kubectl apply -f linstor-s3-snap-restore.yaml

After few moments, you should see the VolumeSnapshotContents marked as ready to use.

$ kubectl config use-context lke
$ kubectl get volumesnapshotcontents.snapshot.storage.k8s.io \
  --no-headers -o custom-columns=":metadata.name,:status.readyToUse"

important-data-restored-content-from-s3   true

Finally, you can create a PVC in the target cluster using the VolumeSnapshotContent restored from S3 as its dataSource using the following example manifest.

$ cat << EOF > restore-important-data-snap-to-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: important-data-restored
  namespace: default
spec:
  storageClassName: linstor-csi-lvm-thin-r3
  accessModes:
  - ReadWriteOnce
  dataSource:
    name: important-data-restored-from-s3
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  resources:
    requests:
      storage: 10Gi
EOF

$ kubectl config use-context lke
$ kubectl apply -f restore-important-data-snap-to-pvc.yaml

With the PersistentVolume and PersistentVolumeClaim now in the target cluster, you can apply your stateful application’s YAML manifest, specifying the newly restored PVC in the target cluster, and should see that your application and data have been migrated.

Concluding Thoughts

Using LINSTOR to abstract away the differences in environment specific block storage means you remain in control of your data, both in terms of how and where it’s used. The need for creating and maintaining separate strategies to account for differences in cloud-provided block storage can be minimized by layering LINSTOR on top of your environment specific storage.

Multi-cloud seems to be the new standard for cloud computing, and being able to move storage between clusters will be a challenge your organization will likely face sooner than later. If you have experienced and solved this already with or without LINSTOR, let me know how you tackled your data migration by reaching out in LINBIT’s community Slack channel.