Reader Bootstrapping:
Amazon Elastic Kubernetes Service (EKS) might be one of the quickest routes an organization can take to run a highly available, fault tolerant, and scalable Kubernetes cluster. EKS handles many of the difficult tasks in managing a Kubernetes cluster for its users; letting them focus more on their applications than their infrastructure. For example, EKS will balance your Kubernetes control plane and worker nodes across AWS availability zones (AZs) in your region, dynamically create EC2 load balancers enabling access to your applications, and even comes with an Elastic Block Store (EBS) storage class for your stateful applications. As great as that all sounds (and is!), there are some gaps that LINBIT SDS® can fill.
Where are these gaps?!
The two main storage offerings available from Amazon for use in EKS are their Elastic Block Store (EBS) and Elastic File System (EFS).
EBS provides block-level storage to EC2 instances – EKS workers in this case – running within the same AZ as the EBS volume. EBS volumes are RWO (read write once) – just like LINBIT SDS and other block storage for Kubernetes – meaning they can only be accessed by one EKS worker instance at a time. This makes EBS a good fit for stateful applications that require performant (low latency/high throughput) block-level storage in AWS. RWO access also means that you’ll have to configure and rely on your application’s built-in replication for fault tolerance and high availability of your application’s stateful data. However, what if your application doesn’t have built-in replication?
EFS provides a filesystem to EC2 instances that can be accessed concurrently and across AZs. In Kubernetes terms, it’s an RWX (read write many) persistent storage option. This acts much like an NFS mounted filesystem. Much like NFS, write performance to EFS suffers from the locking overhead necessary to prevent corruption. EFS is not suitable for use-cases such as OLTP databases, or applications requiring single file access such as Jenkins. EFS is more suited for use-cases such as content repositories, big data, home directories, and media processing.
Thus, if your application requires highly performant storage and doesn’t have the ability to replicate its state on its own, you’re in the gap between EBS and EFS. If you choose EBS, your application could become unavailable in the event of an AZ service outage in your AWS region. If you choose EFS for one of the use cases it wasn’t designed for, you might experience high write latency. LINBIT SDS fills that gap by providing high performance, synchronously replicated, block storage to Kubernetes clusters. With a little EC2 massaging, LINBIT SDS can fill that gap in Amazon EKS.
LINBIT SDS is open-source, and can run on your development laptop just like it can in the cloud. This means your “throw-away dev cluster” can have the same storage backend as your production cluster in AWS, which is another gap LINBIT SDS can help fill.
Preparing EC2 for LINBIT SDS:
LINBIT SDS for Kubernetes deploys a LINSTOR® cluster into your Kubernetes cluster via Helm. LINSTOR will then layer different Linux block technologies such as LVM, VDO, LUKS, and DRBD® on top of one another in order to enable specific features on a volume. The most important layer LINSTOR manages will almost always be DRBD for its in-kernel block replication. In order for LINSTOR to create DRBD devices in our EC2 worker instances, it either needs to ship with a kernel module specific to the Amazon Linux 2.0 kernel (the default Linux distribution for EKS), or the EC2 instances need to have the appropriate kernel-devel packages installed so LINSTOR can compile DRBD for Amazon Linux 2.0 on deployment. As of the publication of this blog, LINSTOR does not ship with the Amazon Linux 2.0 DRBD kernel module. Therefore, we must create a launch template in EC2 for our EKS cluster to use when bootstrapping new EKS worker nodes. LINSTOR also requires an unused block device attached to each EC2 instance that it can use as a storage pool; this can also be done via the EC2 launch template.
Log in to the AWS Management Console and go to the EC2 Dashboard. In the navigation sidebar, you should see a link to “Launch Templates” nested under the “Instances” drop down; follow that link. Then, click the “Create launch template” button in the Launch Template console. Set only the options pictured below, as the rest will be configured elsewhere (depending on how your organization manages EKS).
Name and describe the launch template:
Instance type for launch template:
Set the instance type according to your application’s needs. LINSTOR itself is not resource intensive. Memory utilization for a DRBD resource scales with the size of volume and number of replicas. The formula is roughly 32KiB of memory per 1GiB of storage, multiplied by the number of peers (_other_ nodes with replicas).
Storage settings for launch template:
Volume type for both should be set to General purpose SSD (gp3) for both volumes. The larger volume will be used by LINSTOR when provisioning PVs from its storage classes. Set the size and type according to your requirements. The smaller volume will be used by LINSTOR as an external metadata storage-pool for the DRBD volumes that LINSTOR provisions. The smallest an EBS volume can be is 4GiB, which is enough space for up to 64TiB of LINSTOR provisioned DRBD volumes when replicating 3-ways.
Advanced settings for launch template:
The only advanced settings that need modification is the user data pictured above. This additional user data will be appended to the user data responsible for bootstrapping an EKS worker. For copy and paste ability, here is the user data pictured above:
Content-Type: multipart/mixed; boundary="==BOUNDARY==" MIME-Version: 1.0 --==BOUNDARY== Content-Type: text/x-shellscript; charset="us-ascii" #!/bin/bash echo "Running custom user data script" sudo yum install kernel-devel-`uname -r` -y --==BOUNDARY==--\
Finally, click the “Create template version” button to save the launch template. You should see that your launch template was created successfully. Follow the link to view your launch template and note the “Launch Template ID” which should be formatted like this: lt-0123456789abcdefg. You will need to specify this launch template in the managed node group used by your EKS cluster.
If you use eksctl to create EKS clusters, you can specify your launch template right in your eksctl configuration:
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: linbit-eks
region: us-west-2
managedNodeGroups:
- name: lb-mng-0
launchTemplate:
id: lt-0123456789abcdefg
version: "1"
desiredCapacity: 3
Options to Set for LINSTOR’s Operator in EKS:
With EC2 prepped for LINSTOR, you should be able to follow the LINSTOR User’s Guide to get LINSTOR installed. For convenience, here are the options I would use when deploying LINSTOR via Helm in your EKS cluster:
operator:
controller:
dbConnectionURL: k8s
satelliteSet:
storagePools:
lvmThinPools:
- name: ext-meta-pool
thinVolume: metapool
volumeGroup: ""
devicePaths:
- /dev/nvme1n1
- name: lvm-thin
thinVolume: thinpool
volumeGroup: ""
devicePaths:
- /dev/nvme2n1
kernelModuleInjectionMode: Compile
etcd:
enabled: false
stork:
enabled: false
csi:
enableTopology: true
haController:
replicas: 3
The haController is a key component of LINBIT SDS for Kubernetes. Together with csi.enableTopology, LINSTOR and Kubernetes will make sure your applications are scheduled with your storage more intelligently, and also enable Kubernetes to reschedule your StatefulSet controlled applications in different AZs faster if there ever is an outage.
Also, the additional EBS volumes attached to each EC2 instance that EKS spins up from our launch template will be attached as /dev/nvme1n1 and /dev/nvme2n1, which we’ve set in the LINSTOR operator configuration above. This will cause LINSTOR to automatically prepare those EBS volumes for use as LINSTOR storage pools.
For best performance, and to use the external metadata storage-pool, the following storageClass definitions should be configured in Kubernetes:
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: "linstor-csi-lvm-thin-r1" provisioner: linstor.csi.linbit.com parameters: allowRemoteVolumeAccess: "false" autoPlace: "1" storagePool: "lvm-thin" DrbdOptions/Disk/disk-flushes: "no" DrbdOptions/Disk/md-flushes: "no" DrbdOptions/Net/max-buffers: "10000" property.linstor.csi.linbit.com/StorPoolNameDrbdMeta: "ext-meta-pool" reclaimPolicy: Retain allowVolumeExpansion: true volumeBindingMode: WaitForFirstConsumer --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: "linstor-csi-lvm-thin-r2" provisioner: linstor.csi.linbit.com parameters: allowRemoteVolumeAccess: "false" autoPlace: "2" storagePool: "lvm-thin" DrbdOptions/Disk/disk-flushes: "no" DrbdOptions/Disk/md-flushes: "no" DrbdOptions/Net/max-buffers: "10000" property.linstor.csi.linbit.com/StorPoolNameDrbdMeta: "ext-meta-pool" reclaimPolicy: Retain allowVolumeExpansion: true volumeBindingMode: WaitForFirstConsumer --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: "linstor-csi-lvm-thin-r3" provisioner: linstor.csi.linbit.com parameters: allowRemoteVolumeAccess: "false" autoPlace: "3" storagePool: "lvm-thin" DrbdOptions/Disk/disk-flushes: "no" DrbdOptions/Disk/md-flushes: "no" DrbdOptions/Net/max-buffers: "10000" property.linstor.csi.linbit.com/StorPoolNameDrbdMeta: "ext-meta-pool" reclaimPolicy: Retain allowVolumeExpansion: true volumeBindingMode: WaitForFirstConsumer
If you’ve made it this far and you’re interested in the use case that instigated this blog post, please read the reference architecture I wrote for HA Jenkins deployments in EKS. Or, of course, reach out directly!
