Highly Available NFS Exports with DRBD & Pacemaker

This blog post explains how to configure an NFS server instance in a 3-node high availability (HA) active/passive Linux cluster by using DRBD® and Pacemaker. You can implement this solution with enterprise Linux distributions such as Ubuntu LTS, or Red Hat Enterprise Linux (RHEL).

The instructions in this article have been validated on both RHEL 9 and 10, and Ubuntu 24.04 LTS and 26.04 LTS.

NFS fits many use cases, largely for two reasons: file sharing and centralized storage. File sharing enables multiple systems to access the same files where anyone (or any application) on the network can have access. Centralized storage reduces overall storage costs by sharing centralized storage and eliminating any need for local disk space for each user or application instance.

A few common NFS deployment solutions include:

• Mapping user home directories
• Storing virtual machines (VMs) and persistent data for containers
• Using the ReadWriteMany (RWX) volume access mode in Kubernetes

System preparation requirements

The system preparation requirements for the NFS use case described in this article are:

• You have a 3-node cluster consisting of two diskful nodes for data replication, and one diskless node for DRBD quorum purposes to prevent data divergence.
• Cluster nodes are time-synchronized via Network Time Protocol (NTP).
• The latest version of DRBD is installed and loaded into the kernel on all nodes. DRBD is available from its official GitHub repository, through LINBIT® customer repositories, or the LINBIT PPA for Ubuntu. Refer to the DRBD 9 User Guide for more details.
• Your nodes have a dedicated network link for DRBD replication traffic only. This is a best practice. It is not mandatory.
• The pcs cluster configuration tool is installed on all nodes for editing and managing the Pacemaker configuration.
• Applicable Open Cluster Framework (OCF) resource agents are installed on all nodes (resource-agents* packages).
• NFS server software is installed on all nodes. The NFS server service should be disabled from starting at boot. Pacemaker will manage the NFS server related resources in the cluster, not systemd.
• All cluster nodes can resolve each other’s hostnames over the networks that carry Corosync, DRBD replication, and the virtual IP (VIP) address. Verify through /etc/hosts or your local DNS server.
• Firewalls in use are configured to allow appropriate traffic on all nodes.
  • Corosync: UDP ports 5404, 5405, and 5406.
  • DRBD: TCP port 7788 used in this setup for a single DRBD resource. Refer to the DRBD 9 User Guide for more information.
  • NFS: TCP/UDP port 2049

❗ IMPORTANT: Run nearly all of the commands in this article under the root user account, or preface the commands with sudo for Ubuntu users.

RHEL-specific instructions

If you are using RHEL 9 or 10, take the following steps to prepare your nodes.

Configuring RHEL package repositories

You can install DRBD by using packages in the LINBIT customer repositories. Alternatively, you can compile DRBD and related utilities from source.

Installing DRBD and related utilities on RHEL

Unless you are installing DRBD from source, enter the following command to install the DRBD kernel module and related utilities:

dnf install -y drbd kmod-drbd drbd-pacemaker

The drbd-pacemaker package supplies the ocf:linbit:drbd resource agent at /usr/lib/ocf/resource.d/linbit/drbd.

Verify that the DRBD 9 kernel module is loaded by entering:

modprobe drbd && cat /proc/drbd

Command output should be similar to the following:

version: 9.3.3 (api:2/proto:118-124)
GIT-hash: 97da76040a6b31aaf9e12f1a167e77ca2b3cb43e build by @buildsystem, [...]
Transports (api:22):

Configuring the Red Hat High Availability Add-On

Install the Red Hat Availability Add-On.

The Pacemaker, pcs, and resource-agents packages live in a repository that is not enabled by default, so you need to enable it before the install command in the next section will work.

On RHEL 10, enter:

subscription-manager repos --enable=rhel-10-for-x86_64-highavailability-rpms

On RHEL 9, enter:

subscription-manager repos --enable=rhel-9-for-x86_64-highavailability-rpms

On AlmaLinux and Rocky Linux, enter.

dnf config-manager --set-enabled highavailability

Without this step, a later dnf install -y pacemaker pcs resource-agents iptables command fails with No match for argument for each of the three cluster packages.

💡 TIP: If you are a LINBIT customer, you can skip this step and enable a LINBIT Pacemaker repository on your nodes instead: pacemaker-2 for RHEL 9, or pacemaker-3 for RHEL 10.

Installing Pacemaker and related utilities on RHEL

Enter the following command to install Pacemaker and related utilities:

dnf install -y pacemaker pcs resource-agents iptables

📝 NOTE: In RHEL 10 systems, specifying iptables in the dnf install command will install iptables-nft.

Installing NFS on RHEL

Install NFS and enable the rpcbind service by entering the following commands: ¹

dnf install -y nfs-utils rpcbind
systemctl enable rpcbind --now

Stop and disable the NFS server service, because Pacemaker will control it rather than systemd:

systemctl disable --now nfs-server.service

In some installations the nfs-server.service is disabled already, in which case this command changes nothing.

Configuring the firewall on RHEL

Some cloud and minimal images do not provide firewalld. You can verify whether it exists on a system by entering the following command.

systemctl is-enabled firewalld

If the command returns not-found, there is no firewalld firewall to configure, so skip the rest of this section.

If firewalld is enabled on your nodes, enter the following commands to allow necessary traffic through:

firewall-cmd --permanent --add-service=high-availability
firewall-cmd --permanent --add-service=nfs
firewall-cmd --permanent --add-port=7788/tcp
firewall-cmd --reload

The high-availability service covers the Corosync ports and the pcsd port 2224, the nfs service covers TCP port 2049, and the explicit port rule covers DRBD replication. Because this setup uses NFSv4, which is TCP only, you do not need a UDP rule for port 2049.

Ubuntu-specific instructions

If you are using Ubuntu, take steps in the following sections to prepare your nodes.

Configuring Ubuntu package repositories

You can install DRBD by using packages in the LINBIT customer repositories. Alternatively, you can use the LINBIT PPA. As always, you can also compile DRBD and related utilities from source.

❗IMPORTANT: At the time of writing, the LINBIT PPA does not have all the necessary packages for Ubuntu LTS 26.04 for following the instructions in this article. Use the LINBIT customer package repository on Ubuntu LTS 26.04. Contact the LINBIT team to ask about evaluation access if you are not a LINBIT customer.

Installing DRBD and related utilities on Ubuntu

Unless you are installing DRBD from source, you can install DRBD and related utilities by using software packages. Before installing packages, update repository information and upgrade installed packages.

apt update && apt upgrade -y

❗IMPORTANT: If the upgrade installs a new kernel, reboot the node before continuing so that the new kernel loads before you install the DRBD kernel module by using its Dynamic Kernel Module Support (DKMS) package variant. ²

Install the DRBD kernel module and related packages:

kernel_variant=$(uname -r | sed 's/.*-//')
apt install -y linux-headers-$kernel_variant drbd-dkms drbd-utils

💡 TIP: Here, you set the kernel_variant shell variable to the variant suffix of the running kernel (generic, virtual, lowlatency, and so on). The apt command will then install the correct linux-headers-* meta-package.

Verify the newly installed DRBD 9 kernel module is loaded by entering:

modprobe drbd && cat /proc/drbd

Command output should be similar to the following:

version: 9.3.3 (api:2/proto:118-124)
GIT-hash: 97da76040a6b31aaf9e12f1a167e77ca2b3cb43e build by root@node-0, [...]
Transports (api:22):

Installing Pacemaker and related utilities on Ubuntu

Enter the following command to install Pacemaker and related utilities:

apt install -y pacemaker pcs resource-agents-extra iptables

Installing NFS on Ubuntu

Enter the following command to install NFS and related services and utilities:

apt install -y nfs-kernel-server nfs-common

Disabling NFS services for Pacemaker control

Stop and disable NFS server and its related services in systemd because Pacemaker will control these:

systemctl disable --now nfs-kernel-server.service

📝 NOTE: On Ubuntu, nfs-kernel-server.service is an alias for nfs-server.service, so after you disable it, systemctl is-enabled nfs-kernel-server.service reports alias rather than disabled. The alias status does not indicate that the disable command failed. To verify that the NFS server is disabled and stopped, check the real unit instead:

systemctl is-enabled nfs-server.service
systemctl is-active nfs-server.service

Configuring the firewall in Ubuntu

If ufw is enabled on your nodes, enter the following commands to allow necessary traffic through the firewall:

ufw allow 2049 # NFS
ufw allow 2224/tcp # pcsd
ufw allow 5404:5406/udp # Corosync
ufw allow 7788/tcp # DRBD

Masking the DRBD graceful shutdown service

When Pacemaker manages DRBD, it controls graceful resource demotion during node shutdown. The drbd-graceful-shutdown.service performs the same function. Running both at the same time can cause them to conflict, and potentially trigger an unwanted fence operation. Enter the following command to mask the service.

systemctl mask drbd-graceful-shutdown.service

Creating the initial Pacemaker cluster configuration

After taking preparatory steps specific to your operating system, the rest of the steps are the same for RHEL 9 and 10, Ubuntu 24.04 LTS and 26.04 LTS, and other modern Linux distributions.

First, you will create an initial 3-node Pacemaker cluster configuration. Replace IP addresses and hostnames shown in commands and configurations with those particular to the nodes in your environment.

Enabling and starting the Pacemaker service

On all nodes, enable and start the pcsd.service:

systemctl enable pcsd --now

Setting the Pacemaker user password

On all nodes, set the hacluster user password (replace secretpassword):

echo 'hacluster:secretpassword' | chpasswd

Authenticating the Pacemaker cluster user

On one node, authenticate the hacluster user for each node in the cluster (replace secretpassword):

pcs host auth -u hacluster -p secretpassword \
  node-0 addr=192.168.222.90 \
  node-1 addr=192.168.222.91 \
  node-2 addr=192.168.222.92

Creating the Pacemaker cluster

On one node, create a Pacemaker cluster named linbit-cluster:

# single ring - use when only one network is available
pcs cluster setup --force linbit-cluster \
  node-0 addr=192.168.222.90 \
  node-1 addr=192.168.222.91 \
  node-2 addr=192.168.222.92

# redundant rings - recommended when more than one network is available
pcs cluster setup --force linbit-cluster \
  node-0 addr=192.168.222.90 addr=172.16.0.90 \
  node-1 addr=192.168.222.91 addr=172.16.0.91 \
  node-2 addr=192.168.222.92 addr=172.16.0.92

📝 NOTE: On Ubuntu, installing the pacemaker package starts Corosync and Pacemaker with a default local configuration, so this command first reports two warnings for each node, saying that the host seems to be in a cluster already. The --force option with the pcs cluster setup command allows the command to continue, by destroying the existing default configurations, without needing you to enter a separate pcs cluster destroy command.

Starting Pacemaker and Corosync services

On one node, start the Corosync and Pacemaker services (for all nodes in the cluster). Then enable cluster services to start automatically after a node reboot.

pcs cluster start --all
pcs cluster enable --all

Verifying cluster services and state

Verify that everything has started and is working correctly by entering a pcs status command. It might take a few moments for the cluster to elect a designated coordinator (DC). Before the cluster elects a DC, output can show Current DC: NONE and list every node as UNCLEAN (offline).

After the DC is elected, you should get output similar to the following example:

Cluster name: linbit-cluster

WARNINGS:
error: Resource start-up disabled since no STONITH resources have been defined
error: Either configure some or disable STONITH with the stonith-enabled option
error: NOTE: Clusters with shared data need STONITH to ensure data integrity
error: CIB did not pass schema validation
Configuration invalid (with errors)

Cluster Summary:
  * Stack: corosync (Pacemaker is running)
  * Current DC: node-2 (version [...]) - partition with quorum
  * Last updated: [...]
  * Last change:  [...] by root via cibadmin on node-0
  * 3 nodes configured
  * 0 resource instances configured

Node List:
  * Online: [ node-0 node-1 node-2 ]

Full List of Resources:
  * No resources

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

The WARNINGS block shown here is expected at this point. You have not yet disabled (or configured) STONITH. Pacemaker reports that as an error. A pcs cluster verify command also fails at this point with Error: invalid cib:.

Disabling STONITH

On one node, disable STONITH for the cluster:

pcs property set stonith-enabled=false

The command prints an expected warning:

Warning: Setting property stonith-enabled to false leaves the cluster
with no enabled means to fence nodes, resulting in the cluster not
being able to recover from certain failure conditions

After this, a pcs cluster verify command will complete successfully, and the WARNINGS block will no longer appear in pcs status output.

💡 TIP: While STONITH is highly recommended to prevent split-brains for 2-node clusters, configuring STONITH and fencing is still useful and recommended for 3-node clusters. Because instructions for enabling STONITH and fencing will vary based on the equipment in your environment, and for brevity, STONITH is disabled in these instructions.

However, without STONITH, if a node partially loses cluster connectivity or service monitor operations begin failing, Pacemaker will attempt to stop resources before promoting a replacement node. Those stop operations might time out if the failing node cannot respond. This type of compound failure could leave services stuck, and prevent a successful service failover. With STONITH, a node failing to stop services cleanly is fenced (powered off) allowing the replacement to be promoted safely. This eliminates the possibility of a hung or stuck failover scenario.

Also, without STONITH, if a node completely loses cluster connectivity, this could leave DRBD in a Primary state on the isolated node while also promoting services on a peer. When the isolated node reconnects, if DRBD remained in a Primary role on that node, DRBD will see both nodes are Primary and flag a split-brain scenario which requires manual recovery. With STONITH, an isolated node is fenced (powered off) before the replacement is promoted. This eliminates the possibility of this split-brain scenario.

Creating a logical volume and directory for the NFS share

Before creating a DRBD resource in the cluster, you need to initialize the physical storage device for use with LVM on your diskful nodes.

Run the commands in this section that create LVM objects on the two diskful nodes only, node-0 and node-1. The diskless quorum node has disk none for both volumes in the DRBD configuration later in this article. It does not need a physical volume, volume group, or logical volumes.

On node-0 and node-1, enter the following command:

pvcreate /dev/sdx

Here, “x” in sdx corresponds to the letter identifying your physical storage device.

Next, on node-0 and node-1, create an LVM volume group, named nfs_vg, by entering:

vgcreate nfs_vg /dev/sdx

Next, create the LVM logical volumes that DRBD will consume.

The first logical volume will be for storing NFS stateful connection information and the “tickle” directory used by the portblock OCF resource agent. If the NFS stateful connection is not highly available or otherwise synchronized between cluster nodes, then in some failover cases, it might take a long time for NFS exports to become available. This volume will not hold much data and 20M can be a sufficient size, for testing. ³

The second volume will store data that you will share by using NFS. You can replace the 4 GiB value with a size appropriate for your use, or else use the -l 100%FREE option rather than -L 4G in the command if you want the logical volume to use 100% of available space in the specified volume group.

On node-0 and node-1:

lvcreate -L 20M -n ha_nfs_internal_lv nfs_vg
lvcreate -L 4G -n ha_nfs_exports_lv nfs_vg

After creating the logical volumes, create the directories that will serve as the mount point for your NFS share, and the mount point for cluster internal state information subdirectories. Unlike the LVM commands shown earlier, run these on all three nodes, including the diskless quorum node, because Pacemaker mount points must exist on every node that could host a resource.

mkdir -p /srv/drbd-nfs/exports/HA
mkdir -p /srv/drbd-nfs/internal

The /srv/drbd-nfs/exports directory stays on local disk on every node and acts as a stable parent directory for exported data. The HA subdirectory below it is the mount point for the DRBD-backed file system that holds the data you will export. Keeping the exported file system one level below exports means that you can add further independent exports later, each with its own DRBD volume, file system, fsid value, and exportfs resource primitive, without changing the export path or the client mount commands for an existing export. If you are certain that you will only ever serve a single export from this cluster, you can mount the file system directly at /srv/drbd-nfs/exports and adjust the paths in the remaining commands in this article accordingly.

Configuring DRBD

After preparing your backing storage device and a file system mount point on your nodes, you can next configure DRBD to replicate the storage device across the nodes.

Creating a DRBD resource file

DRBD resource configuration files are located in the /etc/drbd.d/ directory. You need to create resource files on all cluster nodes. You can create a resource file on one node and then use an rsync or scp command to distribute the file to other nodes. Copying the file directly between nodes this way needs root SSH access from one node to the others. If scp fails with Permission denied, either distribute the file from a workstation that can reach all three nodes, or set up SSH keys for the root account between the nodes first. Each DRBD resource that you define in the resource configuration file needs a different TCP port. There is only one defined resource in this configuration, so the configuration uses one TCP port (7788) here. Use the text editor of your choice to create the following DRBD resource file. Change the hostnames and IP addresses to reflect your network configuration.

📝 NOTE: The third cluster node only provides quorum in the cluster. It is not involved in DRBD replication. This type of node is called a DRBD diskless node. In the configuration, this is specified by using the disk none option within the on "node-2" stanza, for each of the volumes, volume 0 and volume 1.

resource "ha_nfs" {
  volume 0 {
    device "/dev/drbd1000";
    disk "/dev/nfs_vg/ha_nfs_internal_lv";
    meta-disk internal;
  }
  volume 1 {
    device "/dev/drbd1001";
    disk "/dev/nfs_vg/ha_nfs_exports_lv";
    meta-disk internal;
   }
  options {
    on-no-quorum suspend-io;
    quorum majority;
  }
  on "node-0" {
    address 192.168.222.90:7788;
    node-id 0;
  }
  on "node-1" {
    address 192.168.222.91:7788;
    node-id 1;
  }
  on "node-2" {
    volume 0 {
      disk none;
    }
    volume 1 {
      disk none;
    }
    address 192.168.222.92:7788;
    node-id 2;
  }
  connection-mesh {
    hosts "node-0" "node-1" "node-2";
  }
}

Initializing the DRBD resource

After creating the DRBD resource configuration file, you need to initialize the DRBD resource. To do this, enter the following commands on both diskful cluster nodes.

📝 NOTE: The first drbdadm command that you run on a node prints the DRBD global usage survey notice. It is informational and is shown only once per node.

drbdadm create-md ha_nfs
drbdadm up ha_nfs

Also, bring the ha_nfs resource up on the diskless quorum node:

drbdadm up ha_nfs

Synchronizing the DRBD resource

Because this is a new file system with no data content, you can save time and skip the initial synchronization.

⚠️ WARNING: Use caution with this command because it can delete any existing data on the logical volume.

Enter the following commands on a diskful node:

drbdadm new-current-uuid --clear-bitmap ha_nfs/0
drbdadm new-current-uuid --clear-bitmap ha_nfs/1

Next, enter and run the following commands on only one of the two diskful cluster nodes. ⁴

drbdadm primary --force ha_nfs
mkfs.ext4 /dev/drbd1000
mkfs.ext4 /dev/drbd1001
mount /dev/drbd1000 /srv/drbd-nfs/internal
mkdir /srv/drbd-nfs/internal/nfs_info_dir
mkdir /srv/drbd-nfs/internal/portblock_tickle_dir
umount /dev/drbd1000

Entering these commands will do a few things:

• Force the node to become primary.
• Create the needed file systems on the DRBD devices.
• Mount the “internal” DRBD device to a mount point.
• Create two “internal” informational directories.
• Unmount the “internal” DRBD device.

The nfs_info_dir directory will store stateful information related to NFS connections. The portblock_tickle_dir directory, also known as the “tickle” directory, will be used by a portblock OCF resource agent to store established TCP connections. Using the portblock resource agent with a “tickle” directory will allow clients to reconnect faster after failover events. See man ocf_heartbeat_portblock for more information.

Creating the tickle directory on every node

The portblock resource agent validates that its tickle_dir path exists on every action it runs, including the initial probe that Pacemaker performs on all nodes. The directory you just created lives on the DRBD-backed file system, which Pacemaker mounts on one node at a time, so on the other nodes the probe would find no directory, return OCF_ERR_INSTALLED, and be recorded as Stopped (Not installed). Pacemaker caches that result and will not start the resource on that node again, which turns the first failover into a complete NFS outage rather than a fast reconnect. ⁵

To prevent that, create the same directory on the local disk of every node, underneath the /srv/drbd-nfs/internal mount point. The path then resolves on every node whether or not the DRBD-backed file system is mounted there. On whichever node holds the mount, the replicated directory is the one in use, so the tickle data is still shared between nodes.

Enter the following command on all three nodes while the DRBD-backed file system is not mounted anywhere, which is the state you are in at this point in these instructions:

mkdir -p /srv/drbd-nfs/internal/portblock_tickle_dir

❗ IMPORTANT: Creating this directory requires that /srv/drbd-nfs/internal is not mounted on the node where you run the command, because otherwise you create it inside the replicated file system again rather than on local disk. Doing it now, before Pacemaker manages the file systems, avoids having to move resources between nodes later to reach each node in turn.

💡 TIP: When making a new file system on a large volume, consider using the -E nodiscard option with the mkfs.ext4 command. Using this option can make the command run faster.

Next, enter lsblk and drbdadm status ha_nfs commands. Output from these commands will be similar to the following.

NAME                        MAJ:MIN  RM  SIZE RO TYPE MOUNTPOINTS
[...]
├─nfs_vg-ha_nfs_internal_lv 253:0     0   20M  0 lvm
│ └─drbd1000                147:1000  0   20M  0 disk
└─nfs_vg-ha_nfs_exports_lv  253:1     0    4G  0 lvm
  └─drbd1001                147:1001  0    4G  0 disk

Device-mapper major and minor numbers are assigned by the system and will differ from this example:

ha_nfs role:Primary
  volume:0 disk:UpToDate open:no
  volume:1 disk:UpToDate open:no
  node-1 role:Secondary
    volume:0 peer-disk:UpToDate
    volume:1 peer-disk:UpToDate
  node-2 role:Secondary
    volume:0 peer-disk:Diskless peer-client:yes
    volume:1 peer-disk:Diskless peer-client:yes

📝 NOTE: Current drbd-utils releases add peer-client:yes for a diskless peer when the command runs outside an interactive terminal, for example, when polled by using an automation tool such as Ansible.

The drbdadm status command should show that DRBD is synchronized and UpToDate. If that is the case, use the following command to change the node from a primary DRBD role to a secondary.

drbdadm secondary ha_nfs

Creating NFS exports and Pacemaker resources

After configuring a DRBD resource as a foundation for a highly available NFS share, you can use Pacemaker to manage the DRBD resource and configure all the components to give you a complete high-availability NFS solution in your cluster. You will again use the pcs command-line tool to incrementally build the Pacemaker configuration.

Configuring the DRBD resource primitive in Pacemaker

First, create the DRBD primitive in the Pacemaker configuration for your cluster, by entering the following commands:

pcs cluster cib drbdconf
pcs -f drbdconf resource create p_drbd_ha_nfs ocf:linbit:drbd \
  drbd_resource=ha_nfs \
  op start interval=0s timeout=40s \
  stop interval=0s timeout=100s \
  monitor interval=31s timeout=20s role=Unpromoted \
  monitor interval=29s timeout=20s role=Promoted
pcs -f drbdconf resource promotable p_drbd_ha_nfs \
  meta promoted-max=1 promoted-node-max=1 clone-max=3 clone-node-max=1 notify=true
pcs cluster cib-push drbdconf

Output from the last command should show CIB updated. The pcs resource promotable command also shows an informational message, Warning: Meta attributes of clone are not validated.

📝 NOTE: The pcs resource promotable command creates a clone resource whose name is the name of the primitive with -clone appended, because promotable resources are a type of clone resource. The p_drbd_ha_nfs primitive therefore becomes the p_drbd_ha_nfs-clone clone resource. Constraints that you create by using pcs commands need to refer to the DRBD resource by the clone name, p_drbd_ha_nfs-clone, rather than the primitive name.

Restricting the DRBD Promoted role to the diskful nodes

A DRBD 9 diskless node can take the Primary role and read and write data through its peer nodes. If you do not configure a constraint that prevents it, Pacemaker can promote the DRBD resource on the diskless quorum node, node-2, and then start the file systems, the NFS server, and the NFS export on that node.

Enter the following commands to restrict promotion of the DRBD resource to your diskful nodes:

pcs cluster cib drbdconf
pcs -f drbdconf constraint location p_drbd_ha_nfs-clone \
  rule role=Promoted score=-INFINITY \
  '#uname ne node-0 and #uname ne node-1'
pcs cluster cib-push drbdconf

📝 NOTE: Pass the rule as a single quoted string, as shown. Splitting it into separate shell arguments also works, but pcs reports Deprecation Warning: Specifying a rule as multiple arguments is deprecated and might be removed in a future release.

Every other Pacemaker resource that you configure in this article is colocated with the Promoted role of the DRBD clone resource, either directly or through a chain of colocation constraints. For this reason, this single constraint keeps all of the NFS resources off the diskless quorum node.

The rule specifies the diskful nodes rather than excluding node-2 by name. Doing it this way ensures that the constraint will still be valid if you add more diskless or quorum-only nodes to the cluster.

Verify the constraint by entering the following command:

pcs constraint config --full

Command output should list a location rule for p_drbd_ha_nfs-clone with a score of -INFINITY for the Promoted role.

Verifying the DRBD resource primitive configuration in Pacemaker

If you enter another pcs status command, the output will show a clone set resource for your DRBD resource. Pacemaker assigns the Promoted role to one of the diskful nodes, so it might not be the first node you listed.

[...]
Full List of Resources:
  * Clone Set: p_drbd_ha_nfs-clone [p_drbd_ha_nfs] (promotable):
    * Promoted: [ node-0 ]
    * Unpromoted: [ node-1 node-2 ]
[...]

You can further verify that Pacemaker is managing your DRBD resource by entering the following command:

drbdadm status ha_nfs

Command output will show the ha_nfs DRBD resource is in a primary role on one of your diskful nodes in your cluster, and in a (replicated) secondary role on the other nodes.

Configuring the file system primitive in Pacemaker

So that the file system that will back the NFS share is only mounted on a node in a primary role for the backing DRBD resource, you need to configure a file system primitive in Pacemaker. The file system primitive is based on the Filesystem OCF resource agent and configures colocation and order constraints to accomplish this.

Because your DRBD resource has two volumes, you will need to configure two file system primitives in Pacemaker.

📝 NOTE: The drbdconf file is a point-in-time copy of the cluster information base (CIB). Pushing the file to the cluster does not refresh it. For this reason, this section and each section that follows begins by recreating the file with a pcs cluster cib drbdconf command. This keeps the working copy aligned with the running cluster and prevents a later push from reverting a change made in the cluster since the copy was taken.

Enter the following commands to configure the file system primitive in Pacemaker, for the NFS stateful information directory:

pcs cluster cib drbdconf
pcs -f drbdconf resource create p_fs_internal ocf:heartbeat:Filesystem \
  device=/dev/drbd1000 \
  directory="/srv/drbd-nfs/internal" \
  fstype=ext4 \
  run_fsck=no \
  op start interval=0s timeout=60s \
  stop interval=0s timeout=60s \
  monitor OCF_CHECK_LEVEL=0 interval=15s timeout=40s
pcs -f drbdconf constraint order \
  promote p_drbd_ha_nfs-clone then start p_fs_internal
pcs -f drbdconf constraint colocation \
  add p_fs_internal with p_drbd_ha_nfs-clone score=INFINITY with-rsc-role=Promoted

Enter the following commands to configure the file system primitive in Pacemaker, for the NFS data share directory:

pcs -f drbdconf resource create p_fs_exports \
  ocf:heartbeat:Filesystem \
  device=/dev/drbd1001 \
  directory="/srv/drbd-nfs/exports/HA" \
  fstype=ext4 \
  run_fsck=no \
  op start interval=0s timeout=60s \
  stop interval=0s timeout=60s \
  monitor OCF_CHECK_LEVEL=0 interval=15s timeout=40s
pcs -f drbdconf constraint order \
  promote p_drbd_ha_nfs-clone then start p_fs_exports
pcs -f drbdconf constraint colocation \
  add p_fs_exports with p_drbd_ha_nfs-clone score=INFINITY with-rsc-role=Promoted

Next, enter the following command to push the updated changes to the Pacemaker CIB:

pcs cluster cib-push drbdconf

Output from this command should show CIB updated.

Verifying the file system primitive configuration in Pacemaker

If you enter another pcs status command, the output will show the two OCF Filesystem resources that you added:

[...]
Full List of Resources:
  * Clone Set: p_drbd_ha_nfs-clone [p_drbd_ha_nfs] (promotable):
    * Promoted: [ node-0 ]
    * Unpromoted: [ node-1 node-2 ]
  * p_fs_internal   (ocf:heartbeat:Filesystem):  Started node-0
  * p_fs_exports    (ocf:heartbeat:Filesystem):  Started node-0
[...]

Enter a df -h command on the node in your cluster that the Filesystem resources are started on, to further verify that Pacemaker has mounted the two file systems, backed by the DRBD devices:

Filesystem                   Size  Used Avail Use% Mounted on
[...]
/dev/drbd1000                 18M   60K   16M   1% /srv/drbd-nfs/internal
/dev/drbd1001                3.9G  8.0K  3.7G   1% /srv/drbd-nfs/exports/HA
[...]

The reported size of the internal volume varies with the space needed to store DRBD internal metadata, so a slightly smaller value than this example is normal.

Configuring the NFS service and exports primitives in Pacemaker

After configuring the Pacemaker resource primitives that will back your HA NFS share, the DRBD volumes, and the file systems that will mount on them, you can configure the Pacemaker primitives that will serve the NFS share in your cluster. The first of these is the NFS service itself.

Configuring the NFS service primitive in Pacemaker

Enter the following commands to configure the NFS service primitive in Pacemaker:

pcs cluster cib drbdconf
pcs -f drbdconf resource create p_nfsserver ocf:heartbeat:nfsserver \
  nfs_shared_infodir=/srv/drbd-nfs/internal/nfs_info_dir \
  nfs_ip=192.168.222.200 \
  op start interval=0s timeout=40s \
  stop interval=0s timeout=20s \
  monitor interval=10s timeout=20s
pcs -f drbdconf constraint colocation \
  add p_nfsserver with p_fs_internal score=INFINITY
pcs -f drbdconf constraint order \
  p_fs_internal then p_nfsserver
pcs -f drbdconf constraint order \
  p_fs_exports then p_nfsserver

The two order constraints start the NFS server only after both file systems mount, p_fs_internal for the stateful information directory and p_fs_exports for the data share directory. Because Pacemaker stops services in the reverse order that they were started, the p_fs_exports constraint also stops the NFS server before that file system unmounts. In a failover, this prevents a system from trying to unmount a file system that nfsd is still serving.

Configuring the NFS exports primitive in Pacemaker

The next Pacemaker resource primitive to configure, related to serving the NFS share in your cluster, is the exportfs OCF resource agent. This resource agent uses the exportfs Linux command to add the NFS export within your cluster. Enter the following commands to configure the NFS exports primitive in Pacemaker.

pcs -f drbdconf resource create p_exportfs ocf:heartbeat:exportfs \
  clientspec=192.168.222.0/24 \
  directory=/srv/drbd-nfs/exports/HA fsid=1 \
  unlock_on_stop=1 options=rw,sync,no_root_squash \
  op start interval=0s timeout=40s \
  stop interval=0s timeout=120s \
  monitor interval=10s timeout=20s
pcs -f drbdconf constraint order \
  p_nfsserver then p_exportfs
pcs -f drbdconf constraint colocation \
  add p_exportfs with p_nfsserver score=INFINITY

📝 NOTE: Change the NFS export file system options shown in this example if you need to, based on your environment or requirements.

After adding these Pacemaker resource primitives, push the changes to your Pacemaker CIB:

pcs cluster cib-push drbdconf

Verifying the NFS service and exports primitives in Pacemaker

If you enter another pcs status command, the output will show the NFS server and NFS exports resources you just added:

[...]
Full List of Resources:
  * Clone Set: p_drbd_ha_nfs-clone [p_drbd_ha_nfs] (promotable):
    * Promoted: [ node-0 ]
    * Unpromoted: [ node-1 node-2 ]
  * p_fs_internal   (ocf:heartbeat:Filesystem):  Started node-0
  * p_fs_exports    (ocf:heartbeat:Filesystem):  Started node-0
  * p_nfsserver (ocf:heartbeat:nfsserver):   Started node-0
  * p_exportfs  (ocf:heartbeat:exportfs):    Started node-0
[...]

Configuring the virtual IP address primitive in Pacemaker

The next Pacemaker resource primitive to configure in your cluster will create and manage the VIP address for your NFS server. Using a VIP address makes the NFS server available within your network from a single, unchanging IP address, regardless of which node in your cluster is currently hosting the service. To add the VIP address to your Pacemaker-managed resources, enter the following commands.

pcs cluster cib drbdconf
pcs -f drbdconf resource create p_virtip ocf:heartbeat:IPaddr2 \
  ip=192.168.222.200 cidr_netmask=24 \
  op monitor interval=20s timeout=20s \
  start interval=0s timeout=20s \
  stop interval=0s timeout=20s
pcs -f drbdconf constraint order \
  p_exportfs then p_virtip
pcs -f drbdconf constraint colocation \
  add p_virtip with p_exportfs score=INFINITY

After adding this Pacemaker resource primitive, push the changes to the Pacemaker CIB:

pcs cluster cib-push drbdconf

Verifying the virtual IP address

As you have done before, enter a pcs status command to verify that the Pacemaker resource primitive you just added is shown in the list of Pacemaker resources:

[...]
Full List of Resources:
  * Clone Set: p_drbd_ha_nfs-clone [p_drbd_ha_nfs] (promotable):
    * Promoted: [ node-0 ]
    * Unpromoted: [ node-1 node-2 ]
  * p_fs_internal   (ocf:heartbeat:Filesystem):  Started node-0
  * p_fs_exports    (ocf:heartbeat:Filesystem):  Started node-0
  * p_nfsserver (ocf:heartbeat:nfsserver):   Started node-0
  * p_exportfs  (ocf:heartbeat:exportfs):    Started node-0
  * p_virtip    (ocf:heartbeat:IPaddr2):     Started node-0
[...]

With the VIP address resource started on a node in your cluster, you can also verify it by entering a ping command. To do this, enter the following command from any node within your system that has a network route to the cluster.

ping -c 4 192.168.222.200

Configuring port block and unblock primitives in Pacemaker

The final Pacemaker resource primitives that you need to configure and add to your setup use the portblock OCF resource agent. Configuring this will allow for faster TCP re-connections for clients on failover. Enter the following commands to add these resource primitives.

❗IMPORTANT: Set status_check=pseudo shown in the following commands for both portblock resource primitives, only on resource-agents releases that have the status_check parameter. See the following note for what your platform provides. ⁶

📝 NOTE: Ubuntu 26.04 (Resolute) provides resource-agents-extra 4.17.0, which has the status_check parameter. RHEL 10 (and AlmaLinux 10) provide a 4.16.0 package that also has it, backported, despite the lower version number. That package also defaults status_check to pseudo for non-promotable resources such as the ones in this article. By setting the parameter explicitly, as shown, you keep the configuration independent of the default value of your distribution, in case it might change with an update. Ubuntu 24.04 (Noble), RHEL 9, and RHEL 8 provide earlier versions without the parameter. On these platforms, leave out the status_check parameter from the commands. The agent already decides whether the resource is running from its own state file rather than from the firewall. This has the same effect as setting status_check=pseudo.

pcs cluster cib drbdconf
pcs -f drbdconf resource create p_pb_block ocf:heartbeat:portblock \
  action=block \
  ip=192.168.222.200 \
  portno=2049 \
  protocol=tcp \
  status_check=pseudo
pcs -f drbdconf constraint order \
  start p_pb_block then p_virtip
pcs -f drbdconf constraint colocation \
  add p_pb_block with p_virtip score=INFINITY
pcs -f drbdconf resource create p_pb_unblock ocf:heartbeat:portblock \
  action=unblock \
  ip=192.168.222.200 \
  portno=2049 \
  tickle_dir="/srv/drbd-nfs/internal/portblock_tickle_dir" \
  protocol=tcp \
  status_check=pseudo \
  op monitor interval=10s timeout=20s
pcs -f drbdconf constraint order \
  start p_virtip then p_pb_unblock
pcs -f drbdconf constraint colocation \
  add p_pb_unblock with p_virtip score=INFINITY
pcs -f drbdconf constraint order \
  p_fs_internal then p_pb_unblock

❗ IMPORTANT: The portblock agent also accepts a reset_local_on_unblock_stop=1 parameter, which earlier versions of this article included. It is left out here to avoid a defect in the resource agent that is fixed upstream but is not yet in any released resource-agents version. ⁷ Before adding it back, verify that the agent installed on every node has the fix:

grep -c 'NftDelete "OUTPUT" "$1" "s" "$3" "$2"' \
  /usr/lib/ocf/resource.d/heartbeat/portblock

If output shows 1, that means the node has the corrected agent. If output shows 0, that means it does not, and you should leave the parameter out.

If pcs resource create shows Warning: Validation result from agent: ERROR: The tickle dir doesn't exist!, do not ignore it. It means the tickle directory is missing on the node where you ran the command, and the earlier step that creates it on every node was skipped or did not take effect.

After adding these Pacemaker resource primitives, push the changes to your Pacemaker CIB:

pcs cluster cib-push drbdconf

Verifying the port block and unblock resources in Pacemaker

Enter a pcs status command to verify that the Pacemaker resource primitives you just added show up within the list of Pacemaker resources:

[...]
Full List of Resources:
  * Clone Set: p_drbd_ha_nfs-clone [p_drbd_ha_nfs] (promotable):
    * Promoted: [ node-0 ]
    * Unpromoted: [ node-1 node-2 ]
  * p_fs_internal   (ocf:heartbeat:Filesystem):  Started node-0
  * p_fs_exports    (ocf:heartbeat:Filesystem):  Started node-0
  * p_nfsserver (ocf:heartbeat:nfsserver):   Started node-0
  * p_exportfs  (ocf:heartbeat:exportfs):    Started node-0
  * p_virtip    (ocf:heartbeat:IPaddr2):     Started node-0
  * p_pb_block  (ocf:heartbeat:portblock):   Started node-0
  * p_pb_unblock    (ocf:heartbeat:portblock):   Started node-0
[...]

You can further verify the NFS port (2049) availability by using a utility such as nmap or telnet from a system on or with a route to the 192.168.222.0/24 network.

With that, you have configured an HA NFS cluster by using DRBD and Pacemaker. The NFS share is ready to be used by clients on your network. You can verify the availability of your NFS share by mounting it from the root user account on any host that is on or has a route to the 192.168.222.0/24 network and has NFS client software.

mkdir -p /mnt/HA
mount 192.168.222.200:/srv/drbd-nfs/exports/HA /mnt/HA

At this point, the root user can read and write to the NFS share from a system that has mounted the share. Configuring the export for access from other user accounts is possible but outside the scope of this article.

Testing failover

Before you put a cluster into production, you should first verify that failover behavior is as expected. For example, the point of the portblock resources is to shorten the interruption that clients see when resources move between nodes.

Before testing failover, start a write loop on the client, against the mounted share, so that you can measure the interruption. Write the loop to a file and run it, rather than pasting it into a shell.

#!/bin/sh
while true; do
    start=$(date +%s.%N)
    if timeout 30 dd if=/dev/zero of=/mnt/HA/testfile bs=4k count=1 \
        conv=fsync >/dev/null 2>&1; then
        rc=OK
    else
        rc=FAIL
    fi
    end=$(date +%s.%N)
    awk -v a="$start" -v b="$end" -v r="$rc" \
        'BEGIN{printf "%.3f %s\n", b-a, r}'
    sleep 0.2
done

With the loop running, you can simulate a failover and move the resources by putting the active node into standby from any node in the cluster:

pcs node standby node-0

Watch pcs status until every resource has started on the other diskful node, then return the first node to service and repeat in the other direction:

pcs node unstandby node-0
pcs node standby node-1
pcs node unstandby node-1

A healthy transition has these characteristics:

No failed writes

Every write in the loop reports OK. One write in the set should take noticeably longer than the rest, while the writes before and after it complete at their usual speed.

A stall confined to one write

Let the loop run for a while before you trigger the failover, so that you have a baseline for how long a normal write takes on your cluster. The length of the interruption depends on your hardware, your network, and the monitor intervals and timeouts you configured. The important characteristics are that the stall is confined to one write and that the loop returns to its baseline immediately afterward. A stall that repeats on every subsequent write, or one that ends only after you intervene, points to a problem.

No residual firewall rules

On every node, both chains of the agent’s own table are empty after the transition completes. A rule left in the INPUT chain blocks new NFS connections, and a rule left in the OUTPUT chain prevents NFS replies. Either one means something went wrong, even if pcs status does not show that there is a problem. To verify there are no residual firewall rules, use a command appropriate for the back end that the portblock agent uses. To check which back end your agent uses, enter the following command.

grep -c NftDelete /usr/lib/ocf/resource.d/heartbeat/portblock

Output of 0 means the agent uses iptables. So does any other output on a node where the nft command is not installed. In that case, enter the following command.

iptables -L -n

Otherwise, for agents that use the nft back end and when the nft command is installed, enter this command.

nft list table inet portblock

A clean cluster state

pcs status --full reports no Failed Resource Actions and no resource annotated (Not installed), and crm_verify -LV exits zero. You should not need to enter a pcs resource cleanup command after a failover.

An unbroken client mount

The client recovers without remounting the share, and a file written before the failover is still readable afterward.

Simplifying constraints with resource sets

The instructions in this article add colocation and order constraints in pairs, one pair at a time, so that you can verify each resource primitive as you add it. You can also express the same relationships in Pacemaker by using resource sets. This can simplify your configuration because two constraints then replace the pairs of constraints that you added throughout the article.

❗IMPORTANT: The resource set constraints shown here are an alternative to the constraints added earlier in this article. Applying both leaves your cluster with duplicate constraints. List the existing constraints and their IDs with a pcs constraint config --full command, and remove each colocation and order constraint that you added in pairs with a pcs constraint delete <constraint_id> command, before adding the constraints shown here. Do not remove the location constraint that restricts the Promoted role to your diskful nodes.

pcs cluster cib setconf
pcs -f setconf constraint colocation set \
  p_fs_internal p_fs_exports p_nfsserver p_exportfs \
  p_pb_block p_virtip p_pb_unblock sequential=false \
  set p_drbd_ha_nfs-clone role=Promoted \
  setoptions score=INFINITY
pcs -f setconf constraint order set \
  p_drbd_ha_nfs-clone action=promote \
  set p_fs_internal p_fs_exports sequential=false \
  set p_nfsserver p_exportfs p_pb_block p_virtip p_pb_unblock \
  setoptions kind=Mandatory
pcs cluster cib-push setconf

❗IMPORTANT: When building a constraint from resource sets, ordering is critical. Pacemaker places the resources in a later set first, and the resources in earlier sets follow them. The set that holds the promotable DRBD clone resource must be the last set in the command. Listing it first tells Pacemaker to place the DRBD resource relative to the other resources, which is the reverse of what you want.

Specifying sequential=false for the two file system resources in the order constraint allows Pacemaker to start them in parallel after it promotes the DRBD clone resource. Resources in the final set are ordered sequentially by default, which reproduces the start order that the earlier sections of this article build up, one constraint at a time.

Verify the result by entering a pcs constraint config --full command, and by moving the resources between nodes with a pcs node standby <node> command.

Conclusion

With DRBD replicating data in real time, and Pacemaker managing resources and services related to your NFS share, if one cluster node fails, the other diskful node takes over automatically.

Some details of setting up this cluster were assumed in the preparation requirements section. If you want more help with those preparatory steps, you can download a more detailed how-to guide from the LINBIT website: NFS High Availability Clustering Using DRBD and Pacemaker on RHEL 9.

Finally, you should consider the values and options used in this article as starting points. Depending on your environment, and the applications and services that you might be running, you might need to adjust things such as monitor intervals, timeouts, on error behaviors, and others. For example, setting cluster-wide resource defaults such as migration-threshold to limit how many failures trigger a resource move, and failure-timeout to control when failure counts reset, can change how Pacemaker handles repeated failures in your environment. If you need help from our experienced team, or have questions about the particulars of setting up a HA NFS within your environment, you can contact the experts at LINBIT.

Changelog

2026-08-26:

• Added a footnote on sizing the internal volume for ext4 versus XFS.
• Corrected the status_check guidance to leave the parameter out on platforms that lack it, rather than installing resource-agents from source.
• Changed the failover firewall check to detect the portblock back end with grep, rather than inferring it from whether firewalld is installed.
• Noted port 111 and rpc-bind handling for using NFS troubleshooting tools from another system.

2026-08-24:

• Reworked the portblock configuration for correctness: removed reset_local_on_unblock_stop=1, set status_check=pseudo on both primitives, and created the tickle directory on every node.
• Updated the RHEL instructions for RHEL 10: added drbd-pacemaker, the HighAvailability repository step, kernel module verification, and disabling nfs-server.service.
• Broadened the firewall section to cover nodes without firewalld.
• Added a “Testing failover” section.
• Corrected the status_check platform guidance and rewrote footnotes 4 through 6.
• Added Ubuntu 26.04 notes on the nfs-server.service alias and the pcs cluster setup cluster-exists warning.
• Clarified the preparation steps: /etc/hosts examples, diskful-only versus all-node commands, root SSH for file distribution, and the STONITH warning in the sample output.
• Refreshed some command output examples
• Fixed a node name typographical error and a single-quoted location constraint rule.
• Added a location constraint that restricts the DRBD Promoted role to the diskful nodes, to keep the NFS resources off the diskless quorum node.
• Shortened Pacemaker resource primitive names.
• Explained the purpose of the exports/HA directory level.
• Explained the clone resource name that pcs resource promotable generates.
• Added steps to re-create the working CIB file after each push to the cluster.
• Added an optional section showing the equivalent constraints as resource sets.

2026-05-29:

• Removed “Pacemaker and Corosync are installed, started, and enabled to start at boot” as a prerequisite. This is shown as steps in the instructions.
• Added steps to mask drbd-graceful-shutdown.service in the RHEL and Ubuntu preparation sections, to prevent it from conflicting with Pacemaker control of DRBD shutdown ordering.
• Added example resource defaults mention in the conclusion.
• Tested instructions and updated for RHEL 10 and Ubuntu 26.04 (Resolute) systems.
• Fixed misleading verification section text in the “Verifying the NFS service and exports primitives” section.
• Added ocf:heartbeat: prefix to portblock resource creation commands for consistency with other commands.
• Added pcs package to RHEL and Ubuntu install commands.
• Fixed tab/space indentation inconsistency in DRBD resource configuration code block.
• Added meta keyword before attribute names in pcs resource promotable because specifying clone and promotable meta attributes without the meta keyword is deprecated.
• Added score= when specifying an INFINITY score with the colocation constraint because specifying score as a standalone value is deprecated in pcs.
• Added status_check=pseudo to the portblock OCF resource agent.
• Made various non-technical language and typographical changes.

2025-05-29:

• Add specific steps for RHEL 9 and Ubuntu 24.04 LTS
• Use pcs rather than crmsh commands
• Add Pacemaker cluster initialization section
• Build Pacemaker configuration step-by-step
• Add more verification steps
• Use updated Pacemaker terminology in primitives and constraints
• Do not use showmount to verify NFS export
• Make language clarity improvements

2024-09-23:

• Technical improvements made to HA architecture.
• Other improvements to technical details of instructions.

2024-07-15:

• LINBIT technical review.

2023-12-04:

• Added user submitted suggestions and LINBIT technical review.

2022-03-09:

• Originally published.