Authors: Joel Zhou & Michael Troutman
Deploying LINBIT SDS into an air-gapped Kubernetes cluster can be useful when your Kubernetes deployment requires persistent, highly available storage. If you are a LINBIT® customer, you can use a combination of container images from LINBIT and some publicly available container images, and then import them into a private Docker registry to carry out the setup.
The YAML configuration files, the container images, and the version numbers that the instructions in this article use are current at the time of writing. However, they might change over time as LINBIT SDS or Kubernetes change over time. We at LINBIT will update these instructions as things might change. If you have any questions or issues following the instructions in this article, you can always contact the experts at LINBIT.
The instructions in this article require you to pull images from a private LINBIT customer-only Docker registry. You will need your LINBIT customer account credentials for this. If you do not have LINBIT customer credentials and are interested in this use case, or LINBIT software in general, you can contact a sales representative to set up an evaluation account.
You will also need to have an existing Docker private registry set up that is accessible to your air-gapped Kubernetes control plane node and any worker nodes that will participate in the storage cluster. If you have not already set this up, you can follow the instructions in this LINBIT knowledge base article.
The examples in this guide use a private Docker registry,
💡 TIP: It will be easier if your Kubernetes control plane node also hosts your private Docker registry. Regardless, you will need to verify that your Kubernetes control plane node and any worker nodes that could run LINBIT SDS pods can find the registry host, whether that host is the control plane node or another host on your air-gapped LAN. You can do this either by using a DNS entry in a DNS service running on the LAN, or else by an entry in each node’s
📝 IMPORTANT: All the LINBIT SDS participating nodes in your Kubernetes cluster, both control plane nodes and worker nodes, will need to have the certificate that you use to encrypt the HTTPS traffic to your private registry added to their system-wide truststores. Refer to the copying and updating the certificate sections of the knowledge base article linked above for instructions on how to do this.
Finally, the instructions in this article will use two utilities, Skopeo and the “command-line JSON processor” (
jq). You will need to install these on the node in your air-gapped LAN that hosts your private Docker registry, and also on the system outside of your LAN that you will use to pull the container images from LINBIT and public Docker registries. Both of these utilities have APT and RPM packages available so you can install them using either an
apt -y install or a
dnf -y install command, depending on the Linux distribution that you are using. The sole purpose of the
jq utility in the instructions is to make some verification output look nicer so you could skip installing it if you want.
The commands shown in this article should be entered from a root user shell.
Downloading the Necessary Container Images
A LINBIT SDS deployment in Kubernetes requires the following container images that you will need to download from an Internet-connected system:
📝 IMPORTANT: Container images from the LINBIT Docker image repository are available to LINBIT customers only. Contact a LINBIT sales representative for an evaluation account if you want trial access to LINBIT software, including LINBIT SDS, and getting-started assistance from LINBIT support staff.
The container images that LINBIT SDS uses might change over time. The images listed above reflect the time of writing this article. For the current list, refer to these pages within the upstream project’s GitHub site:
Using Skopeo to Download Container Images
Rather than using Docker commands to download individual container images and move them one by one into a local private Docker registry, you can use either the Crane or Skopeo tools to do the same on a batch list of images. These instructions will use the Skopeo utility. You can install Skopeo by using a package manager to install the
skopeo package on either a DEB or RPM-based system.
Creating a Batch Lists of Container Images
Create a batch list of container images to download from the LINBIT
drbd.io registry, by using the following command:
cat << EOF > drbd.io-container-images-list.txt drbd.io/drbd-reactor:v1.2.0 drbd.io/drbd-shutdown-guard:v1.0.0 drbd.io/drbd9-amzn2:v9.2.3 drbd.io/drbd9-focal:v9.2.3 drbd.io/drbd9-jammy:v9.2.3 drbd.io/drbd9-rhel7:v9.2.3 drbd.io/drbd9-rhel8:v9.2.3 drbd.io/drbd9-rhel9:v9.2.3 drbd.io/linstor-controller:v1.23.0 drbd.io/linstor-csi:v1.1.0 drbd.io/linstor-k8s-ha-controller:v1.1.4 drbd.io/linstor-operator:v2.1.1 drbd.io/linstor-satellite:v1.23.0 EOF
Depending on the requirements of your applications and services in Kubernetes, you might only need a subset of the operating system container images listed:
You can adjust the batch list of
drbd.io container images above accordingly.
For the other necessary container images that are used in a LINBIT SDS deployment in Kubernetes, create a batch list from publicly available registries, by entering the following command:
cat << EOF > public-repo-container-images-list.txt gcr.io/kubebuilder/kube-rbac-proxy:v0.11.0 quay.io/jetstack/cert-manager-cainjector:v1.12.2 quay.io/jetstack/cert-manager-controller:v1.12.2 quay.io/jetstack/cert-manager-webhook:v1.12.2 quay.io/jetstack/cert-manager-acmesolver:v1.12.2 registry.k8s.io/sig-storage/csi-attacher:v4.3.0 registry.k8s.io/sig-storage/csi-external-health-monitor-controller:v0.8.0 registry.k8s.io/sig-storage/csi-external-health-monitor-controller:v0.9.0 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.8.0 registry.k8s.io/sig-storage/csi-provisioner:v3.5.0 registry.k8s.io/sig-storage/csi-resizer:v1.8.0 registry.k8s.io/sig-storage/csi-snapshotter:v6.2.1 registry.k8s.io/sig-storage/livenessprobe:v2.10.0 EOF
Pulling Container Images to a Local Directory
After creating your batch lists of the necessary container images to deploy LINBIT SDS, you can use Skopeo to pull the images to a local directory.
To pull the necessary publicly available container images to a local directory named
/container-images, enter the following command:
mkdir -p /container-images && \ for image in `cat public-repo-container-images-list.txt`; do \ skopeo sync --src docker --dest dir $image /container-images; done
To pull the necessary container images that are only available to LINBIT customers, enter the following command:
for image in `cat drbd.io-container-images-list.txt`; do \ skopeo sync --src-creds username:password --src docker --dest dir \ $image /container-images; done
📝 IMPORTANT: Downloading images from
drbd.iowill require your LINBIT customer or evaluation account credentials. Replace “username” and “password” above with your LINBIT customer account credentials.
Adding Container Images to A Private Docker Registry
After downloading the container images, bring them to a host that is connected to the same subnet as your air-gapped Kubernetes deployment. You can then import the images into you private Docker registry.
Using Skopeo to Synchronize Container Images
With the container images now on a host in your air-gapped network, you can use Skopeo to import the container images into your private Docker registry. On the host that you copied your downloaded container images to, assuming that the images are located in a directory named
/container-images, enter the following command:
for dir in `ls /container-images`; do \ skopeo sync --src dir --dest docker /container-images/$dir \ my.private-registry.lan:5001; done
Verifying That Container Images Are in Your Private Registry
You can verify that your private Docker registry now has the container images that you synchronized, by entering the following command. Before entering the command, you will need to install the “command-line JSON processor”, by using a package manager to install the
curl https://my.private-registry.lan:5001/v2/_catalog | jq
Output from the command will show a list of container image repositories that contain the container images that you synchronized.
Verifying That You Can Pull Container Images
By default, versions of Kubernetes from 1.20 and later use containerd as a container runtime. Before installing LINBIT SDS, verify that you can use containerd on your control plane node to pull an image from your private Docker registry by entering the following command:
ctr images pull my.private-registry.lan:5001/cert-manager-controller:v1.12.2
Output from the command should show the pulling of the various layers of the container image and eventually show a “done” message, without any errors.
List your local container images by entering a
ctr images ls command. Output should show the container image that you just pulled.
Remove the image by entering the following command:
ctr images rm my.private-registry.lan:5001/cert-manager-controller:v1.12.2
For peace of mind, you can repeat the verification steps on the other participating nodes in your Kubernetes cluster. If you have issues, remember to verify that:
- Nodes in your Kubernetes cluster can resolve your private Docker registry’s hostname to an IP address that they can
ping, or else be able to look up the hostname and IP address by using a local DNS service on the LAN.
- The signing certificate for your private Docker registry is on each node and added to each node’s system-wide truststore.
💡 TIP: If you are using a container runtime other than containerd in your Kubernetes cluster, you can use the Container Runtime Interface (CRI) CLI,
crictl, rather than using
ctrfor verifying your private registry. The CRI CLI also works with the containerd runtime. You will need to edit the CRI CLI configuration file to specify the container runtime endpoint that you are using.
Installing LINBIT SDS
After populating your private Docker registry with the necessary container images, you can install LINBIT SDS into your air-gapped Kubernetes deployment by applying three YAML configuration files.
Creating the YAML Configuration Files
You can create the three YAML configuration files that you will need to deploy LINBIT SDS in Kubernetes by copying the contents of each of the YAML configuration files, pasting them into a text editor, and saving each file as named below.
# Copyright 2022 The cert-manager Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0
apiVersion: piraeus.io/v1 kind: LinstorCluster metadata: name: linstorcluster spec: repository: $PRIVREGHOST:$PRIVREGPORT patches: - target:
apiVersion: v1 kind: Namespace metadata: labels: app.kubernetes.io/name: linbit-sds pod-security.kubernetes.io/enforce: privileged pod-security.kubernetes.io/enforce-version: latest name: linbit-sds
📝 NOTE: You can find the latest version of the LINSTOR Operator YAML at charts.linstor.io. If you download the latest YAML configuration file, you will need to replace the container image references in the YAML file to use the customization variables (
$PRIVREGPORT) before following the next instructions in this article. In the YAML file used above, for example, the lines are:
image: $PRIVREGHOST:$PRIVREGPORT/linstor-operator:v2.1.1 image: $PRIVREGHOST:$PRIVREGPORT/kube-rbac-proxy:v0.11.0
Customizing Configurations to Your Environment
Once you have the three YAML configuration files on your air-gapped Kubernetes control plane node, you can customize them to your environment. There are different ways that you can do this.
If you have the
gettext utility available, you can create environment variables for your private Docker registry hostname and port number, and then use the
envsubst command to replace occurrences of those variables in the configuration files with their values.
The instructions in this guide use a
for loop and a
sed substitution command, as shown below. You can choose a customization method that meets the security or other requirements of your deployment.
Enter the following command to customize the YAML configuration files:
for file in $(ls | grep 'linbit-\(cert-manager\|cluster\|operator\)\.yaml'); \ do sed -i.orig \ -e 's/\$PRIVREGHOST/my\.private-registry\.lan/' \ -e 's/\$PRIVREGPORT/5001/' \ $file; done
Applying the Configurations
After customizing the configurations, you can install LINBIT SDS by applying the YAML configuration files to your Kubernetes deployment.
kubectl apply -f linbit-cert-manager.yaml && \ kubectl wait --for=condition=ready pod --all -n cert-manager --timeout=5m && \ kubectl apply -f linbit-operator.yaml && \ kubectl wait --for=condition=ready pod --all -n linbit-sds --timeout=5m && \ kubectl apply -f linbit-cluster.yaml \ && sleep 2s && # operator needs a moment to create the cluster objects \ kubectl wait --for=condition=ready pod --all -n linbit-sds --timeout=5m && \ echo "LINBIT SDS is up and running in air-gapped Kubernetes!"
Verifying the Installation
To verify that LINBIT SDS is installed into your Kubernetes deployment, you can enter a LINSTOR® client command to list the nodes in your LINBIT SDS cluster:
kubectl exec -it deployments/linstor-controller -n linbit-sds -- linstor n l
Output from the command will be similar to the following:
╭──────────────────────────────────────────────────────────╮ ┊ Node ┊ NodeType ┊ Addresses ┊ State ┊ ╞══════════════════════════════════════════════════════════╡ ┊ kube-0 ┊ SATELLITE ┊ 172.16.145.74:3366 (PLAIN) ┊ Online ┊ ┊ kube-1 ┊ SATELLITE ┊ 172.16.126.70:3366 (PLAIN) ┊ Online ┊ ┊ kube-2 ┊ SATELLITE ┊ 172.16.79.136:3366 (PLAIN) ┊ Online ┊ ╰──────────────────────────────────────────────────────────╯
With LINBIT SDS now deployed in your air-gapped Kubernetes cluster, you can start using it to define persistent storage for your Kubernetes applications and services to consume. If you have not used LINBIT SDS in Kubernetes before, this how-to technical guide, Integrating LINBIT SDS with Kubernetes, a Quick Start Guide , is a good place to start. Because you will have already installed LINBIT SDS in Kubernetes by following the instructions in this article, you can skip to the later sections of the how-to guide for some basic getting-started tasks, for example, creating a StorageClass and a PersistentVolumeClaim.