Deploying LINBIT SDS in an Air-Gapped Kubernetes Cluster

Authors: Joel Zhou & Michael Troutman

—

Deploying LINBIT SDS into an air-gapped Kubernetes cluster can be useful when your Kubernetes deployment requires persistent, highly available storage. If you are a LINBIT® customer, you can use a combination of container images from LINBIT and some publicly available container images, and then import them into a private Docker registry to carry out the setup.

Disclaimer

The YAML configuration files, the container images, and the version numbers that the instructions in this article use are current at the time of writing. However, they might change over time as LINBIT SDS or Kubernetes change over time. We at LINBIT will update these instructions as things might change. If you have any questions or issues following the instructions in this article, you can always contact the experts at LINBIT.

Prerequisites

The instructions in this article require you to pull images from a private LINBIT customer-only Docker registry. You will need your LINBIT customer account credentials for this. If you do not have LINBIT customer credentials and are interested in this use case, or LINBIT software in general, you can contact a sales representative to set up an evaluation account.

You will also need to have an existing Docker private registry set up that is accessible to your air-gapped Kubernetes control plane node and any worker nodes that will participate in the storage cluster. If you have not already set this up, you can follow the instructions in this LINBIT knowledge base article.

The examples in this guide use a private Docker registry, my.private-registry.lan:5001.

💡 TIP: It will be easier if your Kubernetes control plane node also hosts your private Docker registry. Regardless, you will need to verify that your Kubernetes control plane node and any worker nodes that could run LINBIT SDS pods can find the registry host, whether that host is the control plane node or another host on your air-gapped LAN. You can do this either by using a DNS entry in a DNS service running on the LAN, or else by an entry in each node’s /etc/hosts file.

📝 IMPORTANT: All the LINBIT SDS participating nodes in your Kubernetes cluster, both control plane nodes and worker nodes, will need to have the certificate that you use to encrypt the HTTPS traffic to your private registry added to their system-wide truststores. Refer to the copying and updating the certificate sections of the knowledge base article linked above for instructions on how to do this.

Finally, the instructions in this article will use two utilities, Skopeo and the “command-line JSON processor” (jq). You will need to install these on the node in your air-gapped LAN that hosts your private Docker registry, and also on the system outside of your LAN that you will use to pull the container images from LINBIT and public Docker registries. Both of these utilities have APT and RPM packages available so you can install them using either an apt -y install or a dnf -y install command, depending on the Linux distribution that you are using. The sole purpose of the jq utility in the instructions is to make some verification output look nicer so you could skip installing it if you want.

Entering Commands

The commands shown in this article should be entered from a root user shell.

Downloading the Necessary Container Images

A LINBIT SDS deployment in Kubernetes requires the following container images that you will need to download from an Internet-connected system:

• drbd.io/drbd-reactor
• drbd.io/drbd-shutdown-guard
• drbd.io/drbd9-amzn2
• drbd.io/drbd9-focal
• drbd.io/drbd9-jammy
• drbd.io/drbd9-rhel7
• drbd.io/drbd9-rhel8
• drbd.io/drbd9-rhel9
• drbd.io/linstor-controller
• drbd.io/linstor-csi
• drbd.io/linstor-k8s-ha-controller
• drbd.io/linstor-operator
• drbd.io/linstor-satellite
• gcr.io/kubebuilder/kube-rbac-proxy
• k8s.io/livenessprobe
• k8s.io/sig-storage/csi-external-health-monitor-controller
• k8s.io/sig-storage/csi-node-driver-registrar
• k8s.io/sig-storage/csi-provisioner
• k8s.io/sig-storage/csi-resizer
• k8s.io/sig-storage/csi-snapshotter
• k8s.io/sig-storage/livenessprobe
• quay.io/jetstack/cert-manager-acmesolver
• quay.io/jetstack/cert-manager-cainjector
• quay.io/jetstack/cert-manager-controller
• quay.io/jetstack/cert-manager-webhook

📝 IMPORTANT: Container images from the LINBIT Docker image repository are available to LINBIT customers only. Contact a LINBIT sales representative for an evaluation account if you want trial access to LINBIT software, including LINBIT SDS, and getting-started assistance from LINBIT support staff.

The container images that LINBIT SDS uses might change over time. The images listed above reflect the time of writing this article. For the current list, refer to these pages within the upstream project’s GitHub site:

• Piraeus Datastore Images
• Special Interest Group Storage Images

Using Skopeo to Download Container Images

Rather than using Docker commands to download individual container images and move them one by one into a local private Docker registry, you can use either the Crane or Skopeo tools to do the same on a batch list of images. These instructions will use the Skopeo utility. You can install Skopeo by using a package manager to install the skopeo package on either a DEB or RPM-based system.

Creating a Batch List of Container Images

Create a batch list of container images to download from the LINBIT drbd.io registry, by using the following command:

cat << EOF > drbd.io-container-images-list.txt
drbd.io/drbd-reactor:v1.2.0
drbd.io/drbd-shutdown-guard:v1.0.0
drbd.io/drbd9-amzn2:v9.2.3
drbd.io/drbd9-focal:v9.2.3
drbd.io/drbd9-jammy:v9.2.3
drbd.io/drbd9-rhel7:v9.2.3
drbd.io/drbd9-rhel8:v9.2.3
drbd.io/drbd9-rhel9:v9.2.3
drbd.io/linstor-controller:v1.23.0
drbd.io/linstor-csi:v1.1.0
drbd.io/linstor-k8s-ha-controller:v1.1.4
drbd.io/linstor-operator:v2.1.1
drbd.io/linstor-satellite:v1.23.0
EOF

Depending on the requirements of your applications and services in Kubernetes, you might only need a subset of the operating system container images listed:

• drbd.io/drbd9-amzn2
• drbd.io/drbd9-focal
• drbd.io/drbd9-jammy
• drbd.io/drbd9-rhel7
• drbd.io/drbd9-rhel8
• drbd.io/drbd9-rhel9

You can adjust the batch list of drbd.io container images above accordingly.

For the other necessary container images that are used in a LINBIT SDS deployment in Kubernetes, create a batch list from publicly available registries, by entering the following command:

cat << EOF > public-repo-container-images-list.txt
gcr.io/kubebuilder/kube-rbac-proxy:v0.11.0
quay.io/jetstack/cert-manager-cainjector:v1.12.2
quay.io/jetstack/cert-manager-controller:v1.12.2
quay.io/jetstack/cert-manager-webhook:v1.12.2
quay.io/jetstack/cert-manager-acmesolver:v1.12.2
registry.k8s.io/sig-storage/csi-attacher:v4.3.0
registry.k8s.io/sig-storage/csi-external-health-monitor-controller:v0.8.0
registry.k8s.io/sig-storage/csi-external-health-monitor-controller:v0.9.0
registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.8.0
registry.k8s.io/sig-storage/csi-provisioner:v3.5.0
registry.k8s.io/sig-storage/csi-resizer:v1.8.0
registry.k8s.io/sig-storage/csi-snapshotter:v6.2.1
registry.k8s.io/sig-storage/livenessprobe:v2.10.0
EOF

Pulling Container Images to a Local Directory

After creating your batch lists of the necessary container images to deploy LINBIT SDS, you can use Skopeo to pull the images to a local directory.

To pull the necessary publicly available container images to a local directory named /container-images, enter the following command:

mkdir -p /container-images && \
for image in `cat public-repo-container-images-list.txt`; do \
skopeo sync --src docker --dest dir $image /container-images; done

To pull the necessary container images that are only available to LINBIT customers, enter the following command:

for image in `cat drbd.io-container-images-list.txt`; do \
skopeo sync --src-creds username:password --src docker --dest dir \
$image /container-images; done

📝 IMPORTANT: Downloading images from drbd.io will require your LINBIT customer or evaluation account credentials. Replace “username” and “password” above with your LINBIT customer account credentials.

Adding Container Images to A Private Docker Registry

After downloading the container images, bring them to a host that is connected to the same subnet as your air-gapped Kubernetes deployment. You can then import the images into you private Docker registry.

Using Skopeo to Synchronize Container Images

With the container images now on a host in your air-gapped network, you can use Skopeo to import the container images into your private Docker registry. On the host that you copied your downloaded container images to, assuming that the images are located in a directory named /container-images, enter the following command:

for dir in `ls /container-images`; do \
skopeo sync --src dir --dest docker /container-images/$dir \
my.private-registry.lan:5001; done

Verifying That Container Images Are in Your Private Registry

You can verify that your private Docker registry now has the container images that you synchronized, by entering the following command. Before entering the command, you will need to install the “command-line JSON processor”, by using a package manager to install the jq package.

curl https://my.private-registry.lan:5001/v2/_catalog | jq

Output from the command will show a list of container image repositories that contain the container images that you synchronized.

Verifying That You Can Pull Container Images

By default, versions of Kubernetes from 1.20 and later use containerd as a container runtime. Before installing LINBIT SDS, verify that you can use containerd on your control plane node to pull an image from your private Docker registry by entering the following command:

ctr images pull my.private-registry.lan:5001/cert-manager-controller:v1.12.2

Output from the command should show the pulling of the various layers of the container image and eventually show a “done” message, without any errors.

List your local container images by entering a ctr images ls command. Output should show the container image that you just pulled.

Remove the image by entering the following command:

ctr images rm my.private-registry.lan:5001/cert-manager-controller:v1.12.2

For peace of mind, you can repeat the verification steps on the other participating nodes in your Kubernetes cluster. If you have issues, remember to verify that:

• Nodes in your Kubernetes cluster can resolve your private Docker registry’s hostname to an IP address that they can ping, or else be able to look up the hostname and IP address by using a local DNS service on the LAN.
• The signing certificate for your private Docker registry is on each node and added to each node’s system-wide truststore.

💡 TIP: If you are using a container runtime other than containerd in your Kubernetes cluster, you can use the Container Runtime Interface (CRI) CLI, crictl, rather than using ctr for verifying your private registry. The CRI CLI also works with the containerd runtime. You will need to edit the CRI CLI configuration file to specify the container runtime endpoint that you are using.

Installing LINBIT SDS

After populating your private Docker registry with the necessary container images, you can install LINBIT SDS into your air-gapped Kubernetes deployment by applying three YAML configuration files.

Creating the YAML Configuration Files

You can create the three YAML configuration files that you will need to deploy LINBIT SDS in Kubernetes by copying the contents of each of the YAML configuration files, pasting them into a text editor, and saving each file as named below.

linbit-cert-manager.yaml

# Copyright 2022 The cert-manager Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0

This is a snippet. View the Full Code

linbit-operator.yaml

apiVersion: piraeus.io/v1
kind: LinstorCluster
metadata:
  name: linstorcluster
spec:
  repository: $PRIVREGHOST:$PRIVREGPORT
  patches:
    - target:

This is a snippet. View the Full Code

linbit-cluster.yaml

apiVersion: v1
kind: Namespace
metadata:
  labels:
    app.kubernetes.io/name: linbit-sds
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/enforce-version: latest
  name: linbit-sds

This is a snippet. View the Full Code

📝 NOTE: You can find the latest version of the LINSTOR Operator YAML at charts.linstor.io. If you download the latest YAML configuration file, you will need to replace the container image references in the YAML file to use the customization variables ($PRIVREGHOST and $PRIVREGPORT) before following the next instructions in this article. In the YAML file used above, for example, the lines are:

        image: $PRIVREGHOST:$PRIVREGPORT/linstor-operator:v2.1.1
        image: $PRIVREGHOST:$PRIVREGPORT/kube-rbac-proxy:v0.11.0

Customizing Configurations to Your Environment

Once you have the three YAML configuration files on your air-gapped Kubernetes control plane node, you can customize them to your environment. There are different ways that you can do this.

If you have the gettext utility available, you can create environment variables for your private Docker registry hostname and port number, and then use the envsubst command to replace occurrences of those variables in the configuration files with their values.

The instructions in this guide use a for loop and a sed substitution command, as shown below. You can choose a customization method that meets the security or other requirements of your deployment.

Enter the following command to customize the YAML configuration files:

for file in $(ls | grep 'linbit-\(cert-manager\|cluster\|operator\)\.yaml'); \
do sed -i.orig \
-e 's/\$PRIVREGHOST/my\.private-registry\.lan/' \
-e 's/\$PRIVREGPORT/5001/' \
$file; done

Applying the Configurations

After customizing the configurations, you can install LINBIT SDS by applying the YAML configuration files to your Kubernetes deployment.

kubectl apply -f linbit-cert-manager.yaml && \
kubectl wait --for=condition=ready pod --all -n cert-manager --timeout=5m && \
kubectl apply -f linbit-operator.yaml && \
kubectl wait --for=condition=ready pod --all -n linbit-sds --timeout=5m && \
kubectl apply -f linbit-cluster.yaml \ &&
sleep 2s && # operator needs a moment to create the cluster objects \
kubectl wait --for=condition=ready pod --all -n linbit-sds --timeout=5m && \
echo "LINBIT SDS is up and running in air-gapped Kubernetes!"

Verifying the Installation

To verify that LINBIT SDS is installed into your Kubernetes deployment, you can enter a LINSTOR® client command to list the nodes in your LINBIT SDS cluster:

kubectl exec -it deployments/linstor-controller -n linbit-sds -- linstor n l

Output from the command will be similar to the following:

╭──────────────────────────────────────────────────────────╮
┊ Node   ┊ NodeType  ┊ Addresses                  ┊ State  ┊
╞══════════════════════════════════════════════════════════╡
┊ kube-0 ┊ SATELLITE ┊ 172.16.145.74:3366 (PLAIN) ┊ Online ┊
┊ kube-1 ┊ SATELLITE ┊ 172.16.126.70:3366 (PLAIN) ┊ Online ┊
┊ kube-2 ┊ SATELLITE ┊ 172.16.79.136:3366 (PLAIN) ┊ Online ┊
╰──────────────────────────────────────────────────────────╯

Conclusion

With LINBIT SDS now deployed in your air-gapped Kubernetes cluster, you can start using it to define persistent storage for your Kubernetes applications and services to consume. If you have not used LINBIT SDS in Kubernetes before, this how-to technical guide, Integrating LINBIT SDS with Kubernetes, a Quick Start Guide , is a good place to start. Because you will have already installed LINBIT SDS in Kubernetes by following the instructions in this article, you can skip to the later sections of the how-to guide for some basic getting-started tasks, for example, creating a StorageClass and a PersistentVolumeClaim.