LINSTOR is Red Hat Certified for OpenShift Virtualization

LINSTOR® now lists OpenShift Virtualization under Infrastructure features in its Red Hat Ecosystem Catalog listing, next to Container Storage Interface (CSI).

It is a single line, only two words on a catalog spec sheet. In this article, I elaborate on what the certification actually means. Red Hat’s own certification conformance suites were run against the LINSTOR CSI driver running in OpenShift, the results were submitted for review, and a Red Hat CertOps engineer signed off that the driver does what a virtualization storage backend is required to do. This article will cover what was actually tested, what the results were, and a few things I learned about my testing environment, mostly to spice up the announcement post, but also because the testing process took me a while and writing about it helps me justify that. If that’s not interesting to you, and you only care to test the OpenShift Virtualization certified LINSTOR in your Red Hat OpenShift cluster, scroll down to the last few lines of this article where I will mention how you can get started.

The Red Hat OpenShift Virtualization certification coverage

Storage certification for OpenShift Virtualization is two separate tests, defined in chapter 35 of the Red Hat Software Certification Workflow Guide. They check very different things, and both include hundreds of individual tests that all need to pass or be marked as exempt by the Red Hat CertOps engineer reviewer, for successful certification.

The CSI function test

This test is the upstream Kubernetes external-storage end-to-end (e2e) suite, run by using openshift-tests run openshift/csi from Red Hat’s own registry.redhat.io/openshift4/ose-tests-rhel9 image. The e2e suite tests general CSI conformance and covers provisioning, mounting, file system types, block volumes, snapshots, cloning, expansion, topology, and multi-pod access.

The KubeVirt storage checkup

This is the virtualization-specific test suite. It runs the kiagnose checkup from the image shipped in the OpenShift Virtualization CSV. It boots actual virtual machines (VMs) and exercises the operations a VM platform depends on.

Requiring both to pass makes sense when you consider that LINSTOR can be configured in so many different ways, with varying feature sets enabled for each OpenShift storage-class the LINSTOR CSI driver backs. A driver can pass generic CSI conformance and still be a poor fit for VMs, because the things VMs need (live migration, fast clone-from-golden-image, hot plugging a disk into a running guest) are combinations of individual CSI functions. Testing both general CSI conformance and virtualization suitability certifies that users can expect all the bells and whistles that OpenShift Virtualization offers to work when using the LINSTOR CSI driver for their cluster’s persistent storage.

The capabilities that were exercised

One detail worth mentioning about the CSI test suite is that you declare what gets tested. The suite reads a driver manifest, and any capability you do not declare is skipped during the test phase. The manifest is the test plan, and it is what a reviewer processes first.

Here is what was declared for LINSTOR, which is to say here is what was actually put under test:

DriverInfo:
  Name: linstor.csi.linbit.com
  SupportedSizeRange:
    Min: 1Gi
  SupportedFsType:
    ext4: {}
    xfs: {}
  Capabilities:
    persistence: true
    block: true
    fsGroup: true
    exec: true
    multipods: true
    RWX: true
    controllerExpansion: true
    nodeExpansion: true
    onlineExpansion: true
    offlineExpansion: true
    snapshotDataSource: true
    pvcDataSource: true
    topology: true
    capacity: true
    singleNodeVolume: true
    volumeLimits: false

The declared capabilities include raw block volumes, ReadWriteMany (RWX) access, volume expansion both online and offline, snapshots as a data source, PVC-to-PVC cloning, and topology awareness. Undeclared, and therefore skipped, were ntfs, ext3, ReadOnlyMany, ReadWriteOncePod, volume limits, and volume group snapshots. These either are not features of LINSTOR, or else are not required for OpenShift or OpenShift Virtualization certification.

The backing StorageClass was an ordinary two-replica configuration, a placementCount of 2 against an lvm-thin LINSTOR storage pool, with allowRemoteVolumeAccess: true, WaitForFirstConsumer binding, and expansion enabled. Nothing exotic, and nothing you would not deploy in production.

The results

The certified run was on OpenShift 4.20.28 with OpenShift Virtualization 4.20.15, a 3-node compact cluster installed with the agent-based installer, and LINSTOR deployed by the LINSTOR Operator (in two commands). The catalog listing covers the operator across OpenShift 4.12 through 4.22, while the certification run described here was performed on 4.20.28.

The CSI function test finished with 95 passed, 1 failed, 184 skipped. Every applicable test passed except one, which failed “on purpose”. More on that in a moment.

The KubeVirt storage checkup returned status.succeeded: true with an empty failureReason. Every check passed, and the whole checkup took 6 minutes and 34 seconds:

Two of those rows are particularly important for a virtualization platform. The vmLiveMigration check completing means LINSTOR volumes can be handed off between nodes while a guest keeps running. The vmVolumeClone check reporting a cloneType of snapshot means clone-from-golden-image goes through the CSI snapshot path rather than a byte-for-byte host copy, which is the difference between a VM provisioning in seconds and a VM provisioning in minutes.

❗ IMPORTANT: Smart cloning requires a VolumeSnapshotClass for the LINSTOR CSI driver and cloneStrategy: snapshot on the Containerized Data Importer (CDI) StorageProfile. Without both, CDI falls back to a full copy, without notification, and the checkup reports the fallback.

Explaining the one expected test failure

Only a single test failed:

[Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  should concurrently access the single volume from pods on different node

The test suite asked LINSTOR to attach one raw block volume to pods on two different nodes at the same time. LINSTOR refused, with this error:

rpc error: code = FailedPrecondition desc = ControllerPublishVolume failed:
RWX block volume ... does not have the vm.kubevirt.io/name label; RWX block
volumes with allow-two-primaries are only supported for KubeVirt live migration

That refusal is the correct action for LINSTOR. Underneath a RWX block volume, DRBD® has to be put into dual-primary mode, meaning both nodes are allowed to write to the same device simultaneously. There is no coordination layer inside a raw DRBD block device. Two uncoordinated writers on two nodes can corrupt data. File systems that can handle it, such as GFS2 and OCFS2, bring their own cluster-wide locking, while a generic pair of pods does not.

So the LINSTOR CSI driver allows dual-primary access for the one case where the writers are coordinated: KubeVirt live migration. In this case, the source and destination virt-launcher pods hand the guest off under the control of libvirt, and the volume carries the vm.kubevirt.io/name label.

This was submitted as a by-design exception, and the Red Hat reviewing team accepted it. RWX for file system volumes, served through the LINSTOR CSI NFS layer, is supported, and those RWX tests pass.

The test environment

The test suites do not need a large cluster, but they do need a host with enough CPU and memory headroom to boot ten virtual machines at once. The certification runs were done on a nested cluster, with all three OpenShift nodes running as VMs on a single dual E5-2697 v4 host with 72 CPU threads. That was comfortable for both suites, including the parts of the checkup that put every node under load at the same time.

The concurrentVMBoot check is the most demanding test in the KubeVirt storage checkup. It boots 10 VMs concurrently and waits for all of them to become ready within the timeout, vmiTimeout, defined by the test. Memory determines whether all ten guests can be scheduled at once, and, under nested virtualization where I ran my tests, core count determines whether they finish booting before the timeout expires.

After installation, the OpenShift Virtualization operator imports its golden boot images in the background, which creates significant I/O contention against the same storage pool the tests are measuring. Some of my first attempts were on less capable hardware, and letting the import process settle was necessary before running the virtualization storage checkup. In a production cluster, where each of the OpenShift Virtualization hosts are running on their own dedicated cluster node, even with slower storage, this is unlikely to be an issue.

All these factors made it impossible to run the test suites on the reliable but admittedly dated hardware I have available in my test bed without spinning up a full-blown multi-node OpenShift cluster.

Implications for running OpenShift Virtualization

LINSTOR is a supported, Red Hat certified storage layer for VMs on OpenShift. The operations a production VM platform depends on, such as live migration, snapshot-based smart cloning, volume hot plugging, RWX access, and online expansion, are verified by Red Hat’s own suites rather than only by our word.

I’ve already managed to turn, “Hey LINSTOR is now OpenShift Virtualization certified!”, into over 1,000 words, so I will stop now…

If you are new to LINSTOR in Kubernetes and want to try it, reach out to the LINBIT® team to request evaluation access to our customer container registry, drbd.io. Alternatively, you can use the freely available upstream project, Piraeus Operator, to deploy LINSTOR in Kubernetes on your own.