Using VDO with DRBD on RHEL 9

This blog will provide installation and configuration instructions for implementing Red Hat’s Virtual Data Optimizer (VDO) and LINBIT®’s DRBD®.

VDO is a virtual block device driver that provides inline deduplication, compression, and thin provisioning for block storage in Linux. VDO operates within the Linux kernel and is managed by LVM.

DRBD is a virtual block device driver that provides block-level replication between two or more peers over a network. DRBD operates within the Linux kernel and is managed by the drbdadm userland utility.

📝 NOTE: Both DRBD and VDO can also be managed by LINSTOR®, LINBIT’s SDS tool for managing block storage, however that is outside the scope of this blog. For more information see the LINSTOR User’s Guide.

Combining DRBD and VDO by layering DRBD over VDO will provide both data deduplication and synchronous replication of thin-provisioned volumes to your application’s storage. Volumes created within the same volume group will be deduplicated, creating a deduplication domain that spans the entire volume group. Container and virtual machine (VM) images are an excellent use case for such storage topologies, as VMs and containers will share much similar data, and high availability (HA) can be achieved by synchronously replicating virtual disks or container persistent volumes between hypervisors.

Prerequisites and Assumptions

To proceed with this blog you will need to have:

• Red Hat Enterprise Linux (RHEL) 9 installed and registered on two nodes
• A LINBIT customer or evaluation account (contact [email protected])

📝 NOTE: You can also use a RHEL 9 variant such as AlmaLinux or Rocky Linux.

The nodes used must have one or more unused block devices to be used for creating a VDO enabled volume group. You will also need to know your hostnames and IP addresses for configuring DRBD. This blog assumes the following configurations.

Node Configuration Assumptions:

If you intend to leave the firewall enabled between your nodes, you must open the following ports to allow for DRBD’s replication traffic.

Required Ports:

💡 TIP: It is advised to use a dedicated network interface for DRBD. This will separate and isolate the replication traffic from all other traffic on the network. Directly connected network interface cards will both limit latency and reduce the number of failure domains.

Preparing the Systems

The following section will cover the steps needed to prepare the systems for the installation and configuration steps that will follow.

Firewall Rules for DRBD Replication

DRBD replicates writes to peer nodes over the network. If your RHEL systems have firewalld enabled you will need to configure the firewall to allow traffic on that port.

You will use TCP port 7788 for DRBD’s replication traffic. The following command run on both nodes will open the required ports.

# firewall-cmd --add-port=7788/tcp

Installing Packages

The following sections will provide instruction for installation and verification of successful installation of the DRBD and VDO packages.

Installing DRBD

To use DRBD you will need both the DRBD kernel module and DRBD’s userland utilities for management.

If your systems are fully updated to the latest kernel release, you can install DRBD using the following command and skip to the next section of this blog.

# dnf install drbd kmod-drbd -y

If you’re not fully updated and are running on an older kernel version you will need to install the DRBD module specific to your kernel. RHEL supports the Kernel Application Binary Interface (kABI), which will allow for some difference between modules and the running kernel, so if you do not see a matching DRBD kernel module for your specific kernel you should choose the closest DRBD kernel module version without going over.

Use the following process to determine which kernel module is right for your system.

Determine your kernel version using the following command.

# uname -r
5.14.0-70.13.1.el9_0.x86_64

Then, list the available kernel module packages in LINBIT’s repository using the following command.

# dnf search kmod-drbd --showduplicates
LINBIT Packages for drbd-9 - x86_64
94 kB/s | 178 kB     00:01    
=================================================================================
Name Exactly Matched: kmod-drbd
=================================================================================
kmod-drbd-9.1.7_5.14.0_70.13.1.el9_0.x86_64-1.x86_64 : drbd kernel module(s)

In the example above, uname -r outputs the kernel version 5.14.0-70.13.1.el9_0.x86_64. You can see a DRBD kernel module named, kmod-drbd-9.1.7_5.14.0_70.13.1.el9_0.x86_64-1.x86_64, in LINBIT’s repository. Since that is an exact match, that’s the name of the package you’d want to install using the full name, as shown in the command below.

# dnf install drbd-utils kmod-drbd-9.1.7_5.14.0_70.13.1.el9_0.x86_64-1.x86_64 -y

If there was not an exact match to your running kernel, but you saw two packages named:

• kmod-drbd-9.1.7_5.14.0_70.14.1.el9_0.x86_64-1.x86_64
• kmod-drbd-9.1.7_5.14.0_70.12.1.el9_0.x86_64-1.x86_64

You would choose kmod-drbd-9.1.7_5.14.0_70.12.1.el9_0.x86_64-1.x86_64 since that is the closet version to your running kernel without going over in this fictitious example.

Inserting the DRBD Module

With the kernel module installed you can insert it into the running kernel using the following command.

# modprobe drbd

Then, verify it has been properly inserted using the following commands.

# lsmod | grep drbd
drbd                  655360  0
libcrc32c              16384  2 xfs,drbd

# cat /proc/drbd
version: 9.1.7 (api:2/proto:110-121)
GIT-hash: 8eaa30aea61e04fcf4a0cc14d4bea980004f2302 build by @buildsystem, 2022-05-10 07:20:11
Transports (api:18):

Installing VDO

To use VDO you will need both the VDO kernel module and VDO’s userland utilities for management. Install both using the following command.

# dnf install kmod-kvdo vdo lvm2 -y

Inserting the VDO Module

With the kernel module installed you can insert it into the running kernel using the following command.

# modprobe kvdo

Verify the kvdo module is loaded on both systems.

# lsmod | grep vdo

kvdo                  552960  0
uds                   278528  1 kvdo
dm_mod                180224  4 kvdo,dm_log,dm_mirror,dm_bufio

Creating the VDO Backed DRBD Device

To create a VDO backed DRBD device you will need to layer DRBD over top of the VDO device. Your storage will be layered from bottom to top in the following order: physical block device, LVM volume group (VG), VDO device, DRBD device.

This section will cover the configuration and creation of each layer.

Creating the LVM VG and VDO Device

Starting with RHEL 9, VDO can only be managed using LVM tools. You will create the LVM configuration and VDO device in the same section for this reason.

Create a volume group named vdo-vg using /dev/sdb as the physical volume on both nodes.

# vgcreate vdo-vg /dev/sdb
  Physical volume "/dev/sdb" successfully created.
  Volume group "vdo-vg" successfully created

You can then create the VDO volume as a logical volume within your vdo-vg volume group on both nodes.

💡 TIP: VDO’s documentation recommends using a virtual size that is ten times the physical size of your block device for active VM or container storage, or three times the physical size of your block device for object storage.

The following command will create a 20GB physical volume with a 200GB virtual size.

# lvcreate --type vdo -n vdo-lv -L 20G -V 200G vdo-vg/vdo-pool
    The VDO volume can address 16 GB in 8 data slabs, each 2 GB.
    It can grow to address at most 16 TB of physical storage in 8192 slabs.
    If a larger maximum size might be needed, use bigger slabs.
  Logical volume "vdo-lv" created.

📝 NOTE: VDO always reserves at least one entire slab for metadata, and therefore, the reserved slab cannot be used for storing user data.

💡 TIP: The default slab size is 2 GB. A single VDO volume can have up to 8192 slabs. Therefore, the maximum allowed physical storage is 16 TB using the default slab size. When using 32 GB slabs, the maximum allowed physical storage is 256 TB. You can control the slab size by providing the --config 'allocation/vdo_slab_size_mb=size-in-megabytes' option to the lvcreate command.

Creating the DRBD Device

Use a text editor to create the DRBD resource configuration file, /etc/drbd.d/vdo.res, with the following contents on both nodes.

resource vdo {
  device /dev/drbd0;
  disk /dev/vdo-vg/vdo-lv;
  meta-disk internal;
  on rhel9-vdo-alpha {
    address 192.168.122.71:7788;
  }
  on rhel9-vdo-bravo {
    address 192.168.122.72:7788;
  }
}

Create the DRBD metadata for the vdo resource on both nodes.

# drbdadm create-md vdo

Bring the vdo resource up on both nodes.

# drbdadm up vdo

Promote the vdo resource to primary on either node. The first time you promote a new resource to primary you will have to use the --force option since neither node’s data is UpToDate and therefore DRBD cannot determine which node should be the SyncSource or SyncTarget. Whichever node you force into the primary role will become the SyncSource for DRBD’s initial sync.

⚠️ WARNING: If you have existing data and are adding DRBD for replication, be sure to promote the DRBD device on the node with the existing data.

# drbdadm primary --force vdo

Verify that the nodes are synchronizing or reporting an UpToDate disk state using the drbdadm status command on either node.

# drbdadm status
vdo role:Primary
  disk:UpToDate
  rhel9-vdo-bravo role:Secondary
    peer-disk:UpToDate

At this point you can demote the DRBD device back to Secondary on both nodes using the following command on both nodes.

# drbdadm secondary vdo

You now have a DRBD replicated block device that is deduplicated and compressed by VDO and you can use the device in whatever way you see fit. With DRBD’s auto-promote capabilities, you only need to specify the DRBD device, /dev/drbd0, when using utilities such as mkfs and mount on either node. DRBD will handle automatically promoting and demoting nodes for you.

Any operation you perform on the DRBD device on one node will be synchronously replicated to the peer.

Useful Commands and Tools

The following tools can be useful for your VDO and DRBD deployments.

Monitoring VDO’s Deduplication Performance

You can use the vdostats utility to monitor the performance of VDO’s deduplication and compression.

# vdostats
Device                  1k-blocks      Used Available Use% Space saving%
vdo--vg-vdo--pool-vpool   4194304   2888404   1305900  69%            0%

Output from vdostats:

See the man pages for more information and options for the vdostats utility.

Adding High Availability

Your next steps for this deployment might include adding high availability (HA) services that use the VDO backed DRBD devices that you created while following this blog. LINBIT supports using either Pacemaker or DRBD Reactor as the Cluster Resource Manager (CRM) in HA clusters.

As well as adding HA capabilities to your devices, DRBD Reactor also optionally exposes Prometheus metrics for the DRBD device being managed.

DRBD Reactor can be installed from the same LINBIT repository as DRBD 9.

# dnf install drbd-reactor -y

Configuring DRBD Reactor is outside of the scope of this blog. If this is something that interests you, you can find DRBD Reactor in the LINBIT GitHub open source projects page. You can learn more about DRBD Reactor, including how to install and configure it, by referring to the DRBD User’s Guide.

Conclusion

By following this blog you have installed and configured DRBD and VDO for deduplication and replication of your directly attached block devices. How much deduplication and compression of data is possible will vary in your environment and use case. By layering DRBD’s replication over VDO’s deduplication, you add node-level fault tolerance to your system with enterprise grade features, and doing it all by using open source software.

For any questions regarding anything mentioned in this blog or how it might fit into your environment, you can contact the experts at LINBIT for further assistance.