Applying Per-Volume I/O Quality of Service Limits with LINSTOR in Kubernetes

Since the release of the LINSTOR Operator version 2.11.0 and the LINSTOR CSI driver version 1.11.3, both developed by LINBIT®, users can attach I/O Quality of Service (QoS) limits to the persistent volumes that LINSTOR® provisions. Users can specify read and write bandwidth and IOPS limits as StorageClass parameters, and every volume created from that StorageClass inherits those limits. Kubernetes can limit container CPU and memory, but has no built-in mechanism to limit how much disk I/O the persistent volumes of a container can consume.

LINSTOR does not enforce these limits itself, but instead relies on a separate plugin. QoS enforcement happens on the node a pod is scheduled on, and is controlled by an NRI plugin called nri-volume-qos, which you must deploy for LINSTOR QoS settings to work properly.

This article covers how the plugin works, and how you can deploy the plugin, create a QoS-enabled StorageClass, and verify that limits are set and enforced.

An overview of the NRI plugin

Linux enforces block I/O limits through the cgroup v2 io.max controller. ¹ A cgroup io.max file takes one line per block device, keyed by the major:minor numbers of the device:

147:1000 rbps=8388608 wbps=4194304 riops=200 wiops=100

A limit is therefore a property of a cgroup, not of a device. Writing limits somewhere on the storage node would throttle everything touching that device, including LINSTOR replication traffic. To throttle only the workload consuming the volume, the limits have to be written into the cgroup of the specific container that mounts it.

That cgroup does not exist until the container is being created. This is something that neither LINSTOR nor the CSI driver can determine. CSI drivers are called when a volume is provisioned, attached, and mounted, all of which happens before a container is started.

This is where Node Resource Interface (NRI) comes in. NRI is a plugin API that the container runtime, containerd or CRI-O, exposes which lets an external plugin inspect and adjust containers at defined points. nri-volume-qos watches for the CreateContainer event, which triggers after the cgroup of the container has been created but before its process starts. Limits are therefore in place before the first I/O from the container.

The lifecycle of a limit is:

1. You set qos.linbit.com/* parameters on a LINSTOR backed StorageClass.
2. On provisioning, the LINSTOR CSI driver resolves those values and stores them on the LINSTOR resource definition as auxiliary properties.
3. On attach, the driver returns them from its ControllerPublishVolume call, and Kubernetes records them in the status.attachmentMetadata field of the VolumeAttachment object.
4. On container creation, nri-volume-qos reads that metadata, resolves the block device major:minor numbers, and writes the corresponding io.max entries into the cgroup of the container.

The plugin reads five keys from the attachment metadata:

If the device key is missing, the plugin ignores the volume. If you omit any individual limit key, then that setting is unlimited. This means volumes from StorageClasses without QoS parameters are untouched, so you can introduce the plugin to an existing cluster without affecting any current workload.

Prerequisites for deploying the nri-volume-qos plugin

To follow the instructions in this article, you will need:

• A running Kubernetes cluster, version 1.26 or later, with LINSTOR deployed. If you do not yet have LINSTOR in your cluster, you can follow the Integrating LINBIT® SDS with Kubernetes, a Quick Start Guide.
• LINSTOR CSI driver version 1.11.3 or later, which is the release that added the qos.linbit.com/* StorageClass parameters.
• A container runtime with NRI support: containerd 1.7 or later, or CRI-O 1.26 or later.
• Linux kernel 5.14 or later, with cgroup v2 active on your nodes.

The examples that follow were run on a 3-node Kubernetes cluster built on AlmaLinux 9 nodes, running Kubernetes 1.36, containerd 2.3.3, the LINSTOR Operator 2.11.0, LINSTOR 1.34.2, and LINSTOR CSI 1.12.0, backed by thin-provisioned LVM storage pools.

You can confirm that your nodes use cgroup v2 by verifying that the cgroup file system is cgroup2fs:

stat -fc %T /sys/fs/cgroup/

Verify your LINSTOR CSI driver version by inspecting the image that the CSI node DaemonSet runs:

kubectl -n linbit-sds get daemonset linstor-csi-node \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

Enabling NRI in the container runtime

NRI must be enabled in your container runtime before the plugin can register with it. Whether you need to change anything depends on your containerd version. Since containerd 2.0, NRI is enabled in the containerd default configuration. On containerd 1.7, it is disabled by default and you need to turn it on explicitly.

Rather than reading the configuration file, which might not specify the setting at all, check the “running” configuration of the container runtime on each node:

containerd config dump | grep -A2 "io.containerd.nri.v1.nri"

Look for disable = false in the output:

    [plugins.'io.containerd.nri.v1.nri']
      disable = false
      socket_path = '/var/run/nri/nri.sock'

If NRI is disabled, add the following to /etc/containerd/config.toml on each node:

[plugins."io.containerd.nri.v1.nri"]
  disable = false

Then restart containerd:

systemctl restart containerd

📝 NOTE: Restarting containerd does not restart running containers, but it does briefly interrupt the kubelet connection to the runtime. Consider making this change one node at a time outside of peak usage windows, or during a scheduled maintenance window.

Deploying the nri-volume-qos plugin

The plugin runs as a DaemonSet on every node where limits should be enforced. Install it from the Helm chart published to the GitHub Container Registry, pinning a version:

helm install nri-volume-qos oci://ghcr.io/piraeusdatastore/nri-volume-qos \
    --version 0.1.3 --namespace kube-system

Alternatively, you can apply the plain manifest from the project repository:

kubectl apply -f https://raw.githubusercontent.com/piraeusdatastore/nri-volume-qos/v0.1.3/deploy/daemonset.yaml

Either method creates a ServiceAccount, a ClusterRole and binding granting read access to VolumeAttachment objects, and the DaemonSet itself, in the kube-system namespace.

❗ IMPORTANT: The DaemonSet relies on the default DaemonSet tolerations, so it will not schedule onto control-plane nodes or nodes with custom taints unless you add tolerations. If you run workloads on tainted nodes, set the chart’s tolerations value accordingly, or those nodes will not enforce limits.

Confirm that a plugin pod is running on every node:

kubectl -n kube-system get pods -l app.kubernetes.io/name=nri-volume-qos -o wide

Then check the logs of one pod to confirm it registered with the runtime, rather than simply started:

kubectl -n kube-system logs -l app.kubernetes.io/name=nri-volume-qos --tail=10

Successful registration looks like this:

Created plugin 90-qos.linbit.com (nri-volume-qos, handles CreateContainer)
"Starting nri-volume-qos" version="v0.1.3" node="kube-1"
"Waiting for VolumeAttachment informer cache to sync"
"VolumeAttachment informer cache synced"
Registering plugin 90-qos.linbit.com using NRI version v0.12.0...
Configuring plugin 90-qos.linbit.com for runtime containerd/v2.3.3 (NRI version v0.12.0)...
Started plugin 90-qos.linbit.com...

The Configuring plugin ... for runtime line is the one that confirms the runtime accepted the plugin. If NRI is not enabled, the plugin starts but never shows that line.

Creating a QoS-enabled StorageClass

Add the limits you want as qos.linbit.com/* parameters on a StorageClass. Bandwidth parameters accept byte quantities such as 8Mi, and IOPS parameters are integer values.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-csi-lvm-thin-r2-qos
provisioner: linstor.csi.linbit.com
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
parameters:
  linstor.csi.linbit.com/placementCount: "2"
  linstor.csi.linbit.com/storagePool: lvm-thin
  qos.linbit.com/rbps: "8Mi"
  qos.linbit.com/wbps: "4Mi"
  qos.linbit.com/riops: "200"
  qos.linbit.com/wiops: "100"

Apply it to your cluster:

kubectl apply -f linstor-qos-storageclass.yaml

💡 TIP: Use volumeBindingMode: WaitForFirstConsumer on QoS StorageClasses. Limits are resolved when the volume is attached, so binding late keeps the volume placed on the node that runs the pod, rather than placing it first and attaching it over the network from wherever the pod lands.

The limits used here are far below what the backing storage can deliver, which makes their effect easy to see when testing. Choose realistic values for production use.

Verifying that limits reach the volume

Before measuring throughput, you can confirm the limits are propagating all the way down to the node.

Because the StorageClass uses WaitForFirstConsumer, a volume is not provisioned until a pod schedules for it. Create a PVC from the QoS-enabled StorageClass, and a pod to consume it, by entering the following command.

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-limited-volume
spec:
  storageClassName: linstor-csi-lvm-thin-r2-qos
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: qos-verify
spec:
  containers:
    - name: sleeper
      image: busybox
      command: ["sleep", "infinity"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: my-limited-volume
EOF

After the pod is running, there are three places to look, corresponding to the three stages of the path described earlier.

First, confirm that LINSTOR recorded the limits on the resource definition. The CSI driver resolves the StorageClass values into bytes and stores them as auxiliary properties.

kubectl -n linbit-sds exec deploy/linstor-controller -- \
    linstor resource-definition list-properties <pv-name>

╭───────────────────────────────────────────────────────────╮
│ Key                                  │ Value              │
╞═══════════════════════════════════════════════════════════╡
│ Aux/csi-provisioning-completed-by    │ linstor-csi/1.12.0 │
│ Aux/csi.storage.k8s.io/pvc/name      │ my-limited-volume  │
│ Aux/csi.storage.k8s.io/pvc/namespace │ default            │
│ Aux/qos.linbit.com/rbps              │ 8388608            │
│ Aux/qos.linbit.com/riops             │ 200                │
│ Aux/qos.linbit.com/wbps              │ 4194304            │
│ Aux/qos.linbit.com/wiops             │ 100                │
│ DrbdOptions/Resource/quorum          │ majority           │
│ DrbdOptions/auto-verify-alg          │ crct10dif          │
╰───────────────────────────────────────────────────────────╯

Second, confirm that the limits reached the VolumeAttachment object, which is where the plugin reads them from:

kubectl get volumeattachment \
  -o custom-columns=PV:.spec.source.persistentVolumeName,NODE:.spec.nodeName,META:.status.attachmentMetadata

For a volume from a QoS StorageClass, the metadata includes the qos.linbit.com/* keys alongside the usual LINSTOR keys:

PV                                         NODE     META
pvc-d894baa0-c783-4679-9b93-14c4db6887dc   kube-1   map[linstor.csi.linbit.com/device-path:/dev/drbd1000 ... qos.linbit.com/device:/dev/drbd1000 qos.linbit.com/rbps:8388608 qos.linbit.com/riops:200 qos.linbit.com/wbps:4194304 qos.linbit.com/wiops:100]
pvc-a50b8cf4-01cc-433e-a3a9-643be15ae009   kube-1   map[linstor.csi.linbit.com/device-path:/dev/drbd1003 ...]

The second volume above came from a StorageClass without QoS parameters, and lacks qos.linbit.com keys.

Third, confirm that the plugin wrote the limits into the cgroup of the container. Find the container ID, then read io.max from its cgroup on the node running the pod:

kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[0].containerID}'

On that node, using the container ID from the previous command:

CG=$(find /sys/fs/cgroup -type d -name "*<container-id>*" | head -1)
cat "$CG/io.max"

The output has one line per limited device, and major number 147 is DRBD®:

147:1000 rbps=8388608 wbps=4194304 riops=200 wiops=100

Devices belonging to volumes without QoS limits do not appear in this file.

Measuring enforcement with fio

To test enforcement, mount a limited and an unlimited volume in the same pod and run identical fio jobs against both. Both measurements then inherently share a node, a kernel, and a storage subsystem, so the only variable is the QoS limit.

Create a StorageClass for unlimited volumes that references the same storagePool as the QoS restricted StorageClass, and create two PVCs, one from each StorageClass, by entering the following command:

kubectl apply -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-csi-lvm-thin-r2-nolimit
provisioner: linstor.csi.linbit.com
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
parameters:
  linstor.csi.linbit.com/placementCount: "2"
  linstor.csi.linbit.com/storagePool: lvm-thin
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: qos-limited
spec:
  storageClassName: linstor-csi-lvm-thin-r2-qos
  volumeMode: Block
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: qos-unlimited
spec:
  storageClassName: linstor-csi-lvm-thin-r2-nolimit
  volumeMode: Block
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF

Both PVCs specify volumeMode: Block so that fio can write to the raw devices rather than through a file system.

The following Job does that against the two raw block volumes, one from each StorageClass. It runs four separate jobs, keeping reads and writes apart:

apiVersion: batch/v1
kind: Job
metadata:
  name: fio-qos-verify
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: fio
          image: nixery.dev/fio
          command:
            - fio
            - --output-format=normal
            - --ioengine=libaio
            - --direct=1
            - --bs=128k
            - --iodepth=8
            - --runtime=30
            - --time_based
            - --size=512m
            - --name=write-limited
            - --rw=randwrite
            - --filename=/dev/limited
            - --name=write-unlimited
            - --rw=randwrite
            - --filename=/dev/unlimited
            - --name=read-limited
            - --rw=randread
            - --filename=/dev/limited
            - --name=read-unlimited
            - --rw=randread
            - --filename=/dev/unlimited
          volumeDevices:
            - name: limited
              devicePath: /dev/limited
            - name: unlimited
              devicePath: /dev/unlimited
      volumes:
        - name: limited
          persistentVolumeClaim:
            claimName: qos-limited
        - name: unlimited
          persistentVolumeClaim:
            claimName: qos-unlimited

❗ IMPORTANT: Run reads and writes as separate fio jobs, and do not use the --group_reporting fio option.

Wait for the job and read the results:

kubectl wait --for=condition=complete --timeout=600s job/fio-qos-verify
kubectl logs job/fio-qos-verify

Filtering that output down to the throughput lines shows the limits taking effect:

write-limited:   write: IOPS=31,    BW=4095KiB/s (4193kB/s)
write-unlimited: write: IOPS=1174,  BW=147MiB/s (154MB/s)
read-limited:    read:  IOPS=63,    BW=8190KiB/s (8387kB/s)
read-unlimited:  read:  IOPS=17.7k, BW=2214MiB/s (2321MB/s)

The limited volume was capped at 4095 KiB/s of writes against a configured wbps of 4Mi, which is 4096 KiB/s, and 8190 KiB/s of reads against a configured rbps of 8Mi, or 8192 KiB/s. The unlimited volume, on the same node in the same run, wrote at 147 MiB/s.

📝 NOTE: The 2214 MiB/s unlimited read figure is not a measurement of actual disk speed. The test volume was thin-provisioned and largely unwritten, and reads of unallocated thin blocks return immediately without touching the physical disk. It does show that reads were not throttled, but this explains wildly high numbers that you might see when reading from unallocated thin volumes.

The same comparison on file system volumes rather than raw block devices showed limited throughput of roughly 4 MiB/s against approximately 88 MiB/s unlimited. The plugin handles both volume modes.

📝 NOTE: If you end up testing QoS limits on your own by using fio, be careful reading results with --rw=randrw (or any mode that mixes reads and writes), as you will be limited by whichever limit (read or write) is reached first.

Changing the limits on an existing volume

Because the limits are stored on the LINSTOR resource definition, they persist independently of the StorageClass that created the volume. Editing a StorageClass therefore does not retroactively change volumes that already exist.

To change the limits on an existing volume, set the property on its resource definition directly:

kubectl -n linbit-sds exec deploy/linstor-controller -- \
    linstor resource-definition set-property <pv-name> Aux/qos.linbit.com/wbps 1048576

The plugin reads limits from the VolumeAttachment object rather than from LINSTOR directly, so a new value will only be read when a volume has detached and reattached. Restarting the consuming pod is not enough on its own. If a replacement pod starts on the same node before the volume detaches, Kubernetes keeps the existing VolumeAttachment, the ControllerPublishVolume method from the CSI driver is never called again, and the new container is given the old limits.

Stop the workload consuming the volume, either by scaling its controller to zero replicas, or by deleting a standalone pod. After that, verify that the VolumeAttachment for the volume is gone before starting it again.

kubectl delete pod <pod-name>
kubectl get volumeattachment

The attachment is usually removed within a few seconds of the last consuming pod terminating. After it is gone, start the workload again. After the replacement pod starts, io.max in its cgroup will reflect the new limits, and any unchanged values are carried over:

147:1000 rbps=8388608 wbps=1048576 riops=200 wiops=100

❗ IMPORTANT: Limits are read at attach and container-create time, so they cannot change under a running container. When changing a limit, you always need to detach the volume and restart the workload that consumes it.

Use cases for volume I/O QoS and final thoughts

Situations where per-volume limits are worth considering include:

Bounding batch workloads

Workloads such as backup jobs, bulk data imports, and log-processing pipelines are typical high I/O bandwidth consumers. A QoS capped StorageClass limits their impact on latency-sensitive services sharing the same nodes, without needing to use dedicated hardware to isolate them.

Storage tiers

Having several StorageClasses that differ only in their QoS parameters might appeal to service providers who can then offer customers different service tiers on a single physical storage pool.

Multi-tenancy

Tenant-specific capped StorageClasses give each tenant a predictable share of node I/O, and prevent the workload of one tenant workload from degrading that of another.

Predictable performance

An unrestricted volume performs only as well as competing I/O activity on its node allows, so its throughput varies with what its “neighbors” are doing. A QoS restricted volume performs more consistently. This benefits user experience and capacity planning.

 

If you are new to LINSTOR in Kubernetes and want to try it in your own cluster by using container images from the official LINBIT container registry, drbd.io, reach out to the LINBIT team to request evaluation access. Alternatively, you can use the freely available upstream project, the Piraeus Operator, to deploy LINSTOR in Kubernetes.