DRBD’s core functionality is implemented by way of a Linux kernel module. Specifically, DRBD constitutes a driver for a virtual block device, so DRBD is situated right near the bottom of a system’s I/O stack. Because of this, DRBD is extremely flexible and versatile, which makes it a replication solution suitable for adding high availability to just about any application.

DRBD is, by definition and as mandated by the Linux kernel architecture, agnostic of the layers above it. Therefore, it is impossible for DRBD to miraculously add features to upper layers that these do not possess. For example, DRBD cannot auto-detect file system corruption or add active-active clustering capability to file systems like ext3 or XFS.

DRBD includes a set of administration tools which communicate with the kernel module to configure and administer DRBD resources. From top-level to bottom-most these are:

The high-level administration tool of the DRBD-utils program suite. Obtains all DRBD configuration parameters from the configuration file /etc/drbd.conf and acts as a front-end for drbdsetup and drbdmeta. drbdadm has a dry-run mode, invoked with the -d option, that shows which drbdsetup and drbdmeta calls drbdadm would issue without actually calling those commands.

Configures the DRBD module that was loaded into the kernel. All parameters to drbdsetup must be passed on the command line. The separation between drbdadm and drbdsetup allows for maximum flexibility. Most users will rarely need to use drbdsetup directly, if at all.

Allows to create, dump, restore, and modify DRBD metadata structures. Like drbdsetup, most users will only rarely need to use drbdmeta directly.

In DRBD, resource is the collective term that refers to all aspects of a particular replicated data set. These include:

This can be any arbitrary, US-ASCII name not containing white space by which the resource is referred to.

Any resource is a replication group consisting of one or more volumes that share a common replication stream. DRBD ensures write fidelity across all volumes in the resource. Volumes are numbered starting with 0, and there may be up to 65,535 volumes in one resource. A volume contains the replicated data set, and a set of metadata for DRBD internal use.

At the drbdadm level, a volume within a resource can be addressed by the resource name and volume number as resource/volume.

This is a virtual block device managed by DRBD. It has a device major number of 147, and its minor numbers are numbered from 0 onwards, as is customary. Each DRBD device corresponds to a volume in a resource. The associated block device is usually named /dev/drbdX, where X is the device minor number. udev will typically also create symlinks containing the resource name and volume number, as in /dev/drbd/by-res/resource/vol-nr.

A connection is a communication link between two hosts that share a replicated data set. With DRBD 9 each resource can be defined on multiple hosts; with the current versions this requires a full-meshed connection setup between these hosts (that is, each host connected to every other for that resource).

At the drbdadm level, a connection is addressed by the resource and the connection name (the latter defaulting to the peer hostname), like resource:connection.

In DRBD, every resource has a role, which may be Primary or Secondary.

The resource’s role can, of course, be changed, either by manual intervention, by way of some automated algorithm by a cluster management application, or automatically. Changing the resource role from secondary to primary is referred to as promotion, whereas the reverse operation is termed demotion.

DRBD’s hardware and environment requirements and limitations are mentioned below. DRBD can work with just a few KiBs of physical storage and memory, or it can scale up to work with several TiBs of storage and many MiBs of memory.

Since DRBD version 9.2.6, it is possible to encrypt DRBD traffic by using the TLS feature. However, DRBD itself does not contain cryptographic modules. DRBD uses cryptographic modules that are available in the ktls-utils package (used by the tlshd daemon), or that are referenced by the Linux kernel crypto API. In either case, the cryptographic modules that DRBD uses to encrypt traffic will be FIPS compliant, so long as you are using a FIPS mode enabled operating system.

If you have not enabled the TLS feature, then DRBD does not use any cryptographic modules.

In DRBD versions before 9.2.6, it was only possible to use encryption with DRBD if it was implemented in a different block layer, and not by DRBD itself. Linux Unified Key Setup (LUKS) is an example of such an implementation. You can refer to details in the LINSTOR User’s Guide about using LINSTOR as a way that you can layer LUKS below the DRBD layer.

In single-primary mode, a resource is, at any given time, in the primary role on only one cluster member. Since it is guaranteed that only one cluster node manipulates the data at any moment, this mode can be used with any conventional file system (ext3, ext4, XFS, and so on).

Deploying DRBD in single-primary mode is the canonical approach for High-Availability (fail-over capable) clusters.

In dual-primary mode a resource can be in the primary role on two nodes at a time. Since concurrent access to the data is therefore possible, this mode usually requires the use of a shared cluster file system that uses a distributed lock manager. Examples include GFS and OCFS2.

Deploying DRBD in dual-primary mode is the preferred approach for load-balancing clusters which require concurrent data access from two nodes, for example, virtualization environments with a need for live-migration. This mode is disabled by default, and must be enabled explicitly in DRBD’s configuration file.

See Enabling dual-primary mode for information about enabling dual-primary mode for specific resources.

DRBD supports three distinct replication modes, allowing three degrees of replication synchronicity.

Asynchronous replication protocol. Local write operations on the primary node are considered completed as soon as the local disk write has finished, and the replication packet has been placed in the local TCP send buffer. In case of forced fail-over, data loss may occur. The data on the standby node is consistent after fail-over; however, the most recent updates performed prior to the fail-over could be lost. Protocol A is most often used in long distance replication scenarios. When used in combination with DRBD Proxy it makes an effective disaster recovery solution. See Long-distance Replication through DRBD Proxy, for more information.

Memory synchronous (semi-synchronous) replication protocol. Local write operations on the primary node are considered completed as soon as the local disk write has occurred, and the replication packet has reached the peer node. Normally, no writes are lost in case of forced fail-over. However, in case of simultaneous power failure on both nodes and concurrent, irreversible destruction of the primary’s data store, the most recent writes completed on the primary may be lost.

Synchronous replication protocol. Local write operations on the primary node are considered completed only after both the local and the remote disk write(s) have been confirmed. As a result, loss of a single node is guaranteed not to lead to any data loss. Data loss is, of course, inevitable even with this replication protocol if all nodes (respective of their storage subsystems) are irreversibly destroyed at the same time.

By far, the most commonly used replication protocol in DRBD setups is protocol C.

The choice of replication protocol influences two factors of your deployment: protection and latency. Throughput, by contrast, is largely independent of the replication protocol selected.

See Configuring your resource for an example resource configuration which demonstrates replication protocol configuration.

With DRBD 9 it’s possible to have the data stored simultaneously on more than two cluster nodes.

While this has been possible before through stacking, in DRBD 9 this is supported out-of-the-box for (currently) up to 16 nodes. (In practice, using three-, four- or perhaps five-way redundancy through DRBD will make other things the leading cause of downtime.)

The major difference to the stacking solution is that there’s less performance loss, because only one level of data replication is being used.

Prior to DRBD 9, promoting a resource would be done with the drbdadm primary command. With DRBD 9, DRBD will automatically promote a resource to primary role when the auto-promote option is enabled, and one of its volumes is mounted or opened for writing. As soon as all volumes are unmounted or closed, the role of the resource changes back to secondary.

Automatic promotion will only succeed if the cluster state allows it (that is, if an explicit drbdadm primary command would succeed). Otherwise, mounting or opening the device fails as it did prior to DRBD 9.

DRBD supports multiple network transports. As of now two transport implementations are available: TCP and RDMA. Each transport implementation comes as its own kernel module.

DRBD allows configuring multiple paths per connection. The TCP transport uses only one path at a time for a connection, unless you have configured the TCP load balancing feature. The RDMA transport is capable of balancing the network traffic over multiple paths of a single connection. see Configuring multiple paths for more details.

(Re-)synchronization is distinct from device replication. While replication occurs on any write event to a resource in the primary role, synchronization is decoupled from incoming writes. Rather, it affects the device as a whole.

Synchronization is necessary if the replication link has been interrupted for any reason, be it due to failure of the primary node, failure of the secondary node, or interruption of the replication link. Synchronization is efficient in the sense that DRBD does not synchronize modified blocks in the order they were originally written, but in linear order, which has the following consequences:

The service continues to run uninterrupted on the active node, while background synchronization is in progress.

If properly configured, DRBD can detect if the replication network is congested, and suspend replication in this case. In this mode, the primary node “pulls ahead” of the secondary — temporarily going out of sync, but still leaving a consistent copy on the secondary. When more bandwidth becomes available, replication automatically resumes and a background synchronization takes place.

Suspended replication is typically enabled over links with variable bandwidth, such as wide area replication over shared connections between data centers or cloud instances.

See Configuring Congestion Policies and Suspended Replication for details on congestion policies and suspended replication.

Online device verification enables users to do a block-by-block data integrity check between nodes in a very efficient manner.

Note that efficient refers to efficient use of network bandwidth here, and to the fact that verification does not break redundancy in any way. Online verification is still a resource-intensive operation, with a noticeable impact on CPU utilization and load average.

It works by one node (the verification source) sequentially calculating a cryptographic digest of every block stored on the lower-level storage device of a particular resource. DRBD then transmits that digest to the peer node(s) (the verification target(s)), where it is checked against a digest of the local copy of the affected block. If the digests do not match, the block is marked out-of-sync and may later be synchronized. Because DRBD transmits just the digests, not the full blocks, online verification uses network bandwidth very efficiently.

The process is termed online verification because it does not require that the DRBD resource being verified is unused at the time of verification. Therefore, though it does carry a slight performance penalty while it is running, online verification does not cause service interruption or system down time — neither during the verification run nor during subsequent synchronization.

It is a common use case to have online verification managed by the local cron daemon, running it, for example, once a week or once a month. See Using Online Device Verification for information about how to enable, invoke, and automate online verification.

DRBD optionally performs end-to-end message integrity checking using cryptographic message digest algorithms such as MD5, SHA-1, or CRC-32C.

These message digest algorithms are not provided by DRBD, but by the Linux kernel crypto API; DRBD merely uses them. Therefore, DRBD is capable of using any message digest algorithm available in a particular system’s kernel configuration.

With this feature enabled, DRBD generates a message digest of every data block it replicates to the peer, which the peer then uses to verify the integrity of the replication packet. If the replicated block can not be verified against the digest, the connection is dropped and immediately re-established; because of the bitmap the typical result is a retransmission. Therefore, DRBD replication is protected against several error sources, all of which, if unchecked, would potentially lead to data corruption during the replication process:

See Configuring Replication Traffic Integrity Checking for information about how to enable replication traffic integrity checking.

Split brain is a situation where, due to temporary failure of all network links between cluster nodes, and possibly due to intervention by a cluster management software or human error, both nodes switched to the Primary role while disconnected. This is a potentially harmful state, as it implies that modifications to the data might have been made on either node, without having been replicated to the peer. Therefore, it is likely in this situation that two diverging sets of data have been created, which cannot be trivially merged.

DRBD split brain is distinct from cluster split brain, which is the loss of all connectivity between hosts managed by a distributed cluster management application such as Pacemaker. To avoid confusion, this guide uses the following convention:

DRBD allows for automatic operator notification (by email or other means) when it detects split brain. See Split Brain Notification for details on how to configure this feature.

While the recommended course of action in this scenario is to manually resolve the split brain and then eliminate its root cause, it may be desirable, in some cases, to automate the process. DRBD has several resolution algorithms available for doing so:

Whether or not automatic split brain recovery is acceptable depends largely on the individual application. Consider the example of DRBD hosting a database. The discard modifications from host with fewer changes approach may be fine for a web application click-through database. By contrast, it may be totally unacceptable to automatically discard any modifications made to a financial database, requiring manual recovery in any split brain event. Consider your application’s requirements carefully before enabling automatic split brain recovery.

Refer to Automatic Split Brain Recovery Policies for details on configuring DRBD’s automatic split brain recovery policies.

When local block devices such as hard drives or RAID logical disks have write caching enabled, writes to these devices are considered completed as soon as they have reached the volatile cache. Controller manufacturers typically refer to this as write-back mode, the opposite being write-through. If a power outage occurs on a controller in write-back mode, the last writes are never committed to the disk, potentially causing data loss.

To counteract this, DRBD makes use of disk flushes. A disk flush is a write operation that completes only when the associated data has been committed to stable (non-volatile) storage — that is to say, it has effectively been written to disk, rather than to the cache. DRBD uses disk flushes for write operations both to its replicated data set and to its metadata. In effect, DRBD circumvents the write cache in situations it deems necessary, as in activity log updates or enforcement of implicit write-after-write dependencies. This means additional reliability even in the face of power failure.

It is important to understand that DRBD can use disk flushes only when layered on top of backing devices that support them. Most reasonably recent kernels support disk flushes for most SCSI and SATA devices. Linux software RAID (md) supports disk flushes for RAID-1 provided that all component devices support them too. The same is true for device-mapper devices (LVM2, dm-raid, multipath).

Controllers with battery-backed write cache (BBWC) use a battery to back up their volatile storage. On such devices, when power is restored after an outage, the controller flushes all pending writes out to disk from the battery-backed cache, ensuring that all writes committed to the volatile cache are actually transferred to stable storage. When running DRBD on top of such devices, it may be acceptable to disable disk flushes, thereby improving DRBD’s write performance. See Disabling Backing Device Flushes for details.

Trim and Discard are two names for the same feature: a request to a storage system, telling it that some data range is not being used anymore^[2] and can be erased internally.
This call originates in Flash-based storages (SSDs, FusionIO cards, and so on), which cannot easily rewrite a sector but instead have to erase and write the (new) data again (incurring some latency cost). For more details, see for example, the wikipedia page.

Since 8.4.3 DRBD includes support for Trim/Discard. You don’t need to configure or enable anything; if DRBD detects that the local (underlying) storage system allows using these commands, it will transparently enable them and pass such requests through.

The effect is that for example, a recent-enough mkfs.ext4 on a multi-TB volume can shorten the initial sync time to a few seconds to minutes – just by telling DRBD (which will relay that information to all connected nodes) that most/all of the storage is now to be seen as invalidated.

Nodes that connect to that resource later on will not have seen the Trim/Discard requests, and will therefore start a full resync; depending on kernel version and file system a call to fstrim might give the wanted result, though.

If a hard disk that is used as a backing block device for DRBD on one of the nodes fails, DRBD may either pass on the I/O error to the upper layer (usually the file system) or it can mask I/O errors from upper layers.

If DRBD is configured to pass on I/O errors, any such errors occurring on the lower-level device are transparently passed to upper I/O layers. Therefore, it is left to upper layers to deal with such errors (this may result in a file system being remounted read-only, for example). This strategy does not ensure service continuity, and is therefore not recommended for most users.

If DRBD is configured to detach on lower-level I/O error, DRBD will do so, automatically, upon occurrence of the first lower-level I/O error. The I/O error is masked from upper layers while DRBD transparently fetches the affected block from a peer node, over the network. From then onwards, DRBD is said to operate in diskless mode, and carries out all subsequent I/O operations, read and write, on the peer node(s) only. Performance in this mode will be reduced, but the service continues without interruption, and can be moved to the peer node in a deliberate fashion at a convenient time.

See Configuring I/O Error Handling Strategies for information about configuring I/O error handling strategies for DRBD.

DRBD distinguishes between inconsistent and outdated data. Inconsistent data is data that cannot be expected to be accessible and useful in any manner. The prime example for this is data on a node that is currently the target of an ongoing synchronization. Data on such a node is part obsolete, part up to date, and impossible to identify as either. Therefore, for example, if the device holds a file system (as is commonly the case), that file system would be unexpected to mount or even pass an automatic file system check.

Outdated data, by contrast, is data on a secondary node that is consistent, but no longer in sync with the primary node. This would occur in any interruption of the replication link, whether temporary or permanent. Data on an outdated, disconnected secondary node is expected to be clean, but it reflects a state of the peer node some time past. To avoid services using outdated data, DRBD disallows promoting a resource that is in the outdated state.

DRBD has interfaces that allow an external application to outdate a secondary node as soon as a network interruption occurs. DRBD will then refuse to switch the node to the primary role, preventing applications from using the outdated data. A complete implementation of this functionality exists for the Pacemaker cluster management framework (where it uses a communication channel separate from the DRBD replication link). However, the interfaces are generic and may be easily used by any other cluster management application.

Whenever an outdated resource has its replication link re-established, its outdated flag is automatically cleared. A background synchronization then follows.

When using three-way replication, DRBD adds a third node to an existing 2-node cluster and replicates data to that node, where it can be used for backup and disaster recovery purposes. This type of configuration generally involves Long-distance Replication through DRBD Proxy.

Three-way replication works by adding another, stacked DRBD resource on top of the existing resource holding your production data, as seen in this illustration:

The stacked resource is replicated using asynchronous replication (DRBD protocol A), whereas the production data would usually make use of synchronous replication (DRBD protocol C).

Three-way replication can be used permanently, where the third node is continuously updated with data from the production cluster. Alternatively, it may also be employed on demand, where the production cluster is normally disconnected from the backup site, and site-to-site synchronization is performed on a regular basis, for example by running a nightly cron job.

DRBD’s protocol A is asynchronous, but the writing application will block as soon as the socket output buffer is full (see the sndbuf-size option in the man page of drbd.conf). In that event, the writing application has to wait until some of the data written runs off through a possibly small bandwidth network link.

The average write bandwidth is limited by available bandwidth of the network link. Write bursts can only be handled gracefully if they fit into the limited socket output buffer.

You can mitigate this by DRBD Proxy’s buffering mechanism. DRBD Proxy will place changed data from the DRBD device on the primary node into its buffers. DRBD Proxy’s buffer size is freely configurable, only limited by the address room size and available physical RAM.

Optionally DRBD Proxy can be configured to compress and decompress the data it forwards. Compression and decompression of DRBD’s data packets might slightly increase latency. However, when the bandwidth of the network link is the limiting factor, the gain in shortening transmit time outweighs the added latency of compression and decompression.

Compression and decompression were implemented with multi core SMP systems in mind, and can use multiple CPU cores.

The fact that most block I/O data compresses very well and therefore the effective bandwidth increases justifies the use of the DRBD Proxy even with DRBD protocols B and C.

See Using DRBD Proxy for information about configuring DRBD Proxy.

Truck-based replication, also known as disk shipping, is a means of preseeding a remote site with data to be replicated, by physically shipping storage media to the remote site. This is particularly suited for situations where

In such situations, without truck-based replication, DRBD would require a very long initial device synchronization (on the order of weeks, months, or years). Truck based replication allows shipping a data seed to the remote site, and so drastically reduces the initial synchronization time. See Using truck based replication for details on this use case.

A somewhat special use case for DRBD is the floating peers configuration. In floating peer setups, DRBD peers are not tied to specific named hosts (as in conventional configurations), but instead have the ability to float between several hosts. In such a configuration, DRBD identifies peers by IP address, rather than by host name.

For more information about managing floating peer configurations, see Configuring DRBD to Replicate Between Two SAN-backed Pacemaker Clusters.

If your company’s policy says that 3-way redundancy is needed, you need at least 3 servers for your setup.

Now, as your storage demands grow, you will encounter the need for additional servers. Rather than having to buy 3 more servers at the same time, you can rebalance your data across a single additional node.

In the figure above you can see the before and after states: from 3 nodes with three 25TiB volumes each (for a net 75TiB), to 4 nodes, with net 100TiB.

DRBD 9 makes it possible to do an online, live migration of the data; please see Data Rebalancing for the exact steps needed.

With the multiple-peer feature of DRBD, several interesting use cases have been added, for example the DRBD client.

The basic idea is that the DRBD back end can consist of three, four, or more nodes (depending on the policy of required redundancy); but, as DRBD 9 can connect more nodes than that. DRBD works then as a storage access protocol in addition to storage replication.

All write requests executed on a primary DRBD client gets shipped to all nodes equipped with storage. Read requests are only shipped to one of the server nodes. The DRBD client will evenly distribute the read requests among all available server nodes.

See Permanently Diskless Nodes for more information.

DRBD quorum is a feature that can help you avoid split-brain situations and data divergence in high-availability clusters. By using DRBD quorum, you do not have to resort to using fencing or STONITH solutions, although you can also use these if you want to. DRBD quorum requires at least three nodes but a third node which acts as an arbitrator can be diskless. An arbitrator node does not need to have the same hardware specifications as diskful storage nodes in your cluster. For example, a low-powered single-board computer such as a Raspberry Pi might even be sufficient.

The functional concept behind DRBD quorum is that a cluster node can only modify the DRBD replicated data set if the number of DRBD-running nodes for the data set that the node can communicate with, including the node itself, meets the requirement that you specify when you enable the quorum option. For most configurations, this will be a majority number of nodes. A majority number is more than half of the total number of DRBD-running nodes for the data set in the cluster. By only allowing data writes on a node that has network access to more than half the nodes in a given partition, including the node itself, the DRBD quorum feature helps you avoid a situation that would create a diverging data set.

It is not however the case that you always need two or more running nodes in a 3-node cluster for a DRBD primary node to be able to write to the data set. There is an exception, for example, if you disconnect all secondary nodes gracefully, then DRBD will mark their data as outdated as they leave the cluster. In this way, a single remaining DRBD primary node would “know” that it is safe to continue to write data. This situation could arise, for example, while performing system maintenance on nodes in your cluster. This way, you can maintain nodes in your cluster without causing downtime for your applications and services.

Using DRBD quorum is compatible with running a Pacemaker cluster. Pacemaker gets informed about quorum or loss-of-quorum through the master score of the DRBD resource.

There are different options and behaviors that you can configure related to the DRBD quorum feature, such as how you define quorum in your cluster and what actions DRBD might take when a node loses quorum. Refer to the Configuring Quorum section for information about this.

DRBD runs all its necessary resync operations in parallel so that nodes are reintegrated with up-to-date data as soon as possible. This works well when there is one DRBD resource per backing disk.

However, when DRBD resources share a physical disk (or when a single resource spans multiple volumes), resyncing these resources (or volumes) in parallel results in a nonlinear access pattern. Hard disks perform much better with a linear access pattern. For such cases you can serialize resyncs using the resync-after keyword within a disk section of a DRBD resource configuration file.

See here for an example.

In many scenarios it is useful to combine DRBD with a failover cluster resource manager. DRBD can integrate with a cluster resource manager (CRM) such as DRBD Reactor and its promoter plug-in, or Pacemaker, to create failover clusters.

DRBD Reactor is an open source tool that monitors DRBD events and reacts to them. Its promoter plug-in manages services using systemd unit files or OCF resource agents. Since DRBD Reactor solely relies on DRBD’s cluster communication, no configuration for its own communication is needed.

DRBD Reactor requires that quorum is enabled on the DRBD resources it is monitoring, so a failover cluster must have a minimum of three nodes. A limitation is that it supports ordering of services only for collocated services. One of its advantages is that it makes possible fully automatic recovery of clusters after a temporary network failure. This, together with its simplicity, make it the recommended failover cluster manager. Furthermore, DRBD Reactor is perfectly suitable for deployments on clouds as it needs no STONITH or redundant networks in deployments with three or more nodes (for quorum).

Pacemaker is the longest available open source cluster resource manager for high-availability clusters. It requires its own communication layer (Corosync) and it requires STONITH to deal with various scenarios. STONITH might require dedicated hardware and it can increase the impact radius of a service failure. Pacemaker probably has the most flexible system to express resource location and ordering constraints. However, with this flexibility, setups can become complex.

Finally, there are also proprietary solutions for failover clusters that work with DRBD, such as SIOS LifeKeeper for Linux, HPE Serviceguard for Linux, and Veritas Cluster Server.

Veritas Cluster Server (or Veritas InfoScale Availability) is a commercial alternative to the Pacemaker open source software. In case you need to integrate DRBD resources into a VCS setup please see the README in drbd-utils/scripts/VCS on github.

LINBIT, the DRBD project’s sponsor company, provides binary packages to its commercial support customers. These packages are available through repositories and package manager commands (for example, apt, dnf), and when reasonable through LINBIT’s Docker registry. Packages and images from these sources are considered “official” builds.

These builds are available for the following distributions:

Refer to the LINBIT Kernel Module Signing for Secure Boot section for information about which specific DRBD kernel modules have signed packages for which distributions.

Packages for some other distributions are built as well, but don’t receive as much testing.

LINBIT releases binary builds in parallel with any new DRBD source release.

Package installation on RPM-based systems (SLES, RHEL, AlmaLinux) is done by simply using dnf install (for new installations) or dnf update (for upgrades).

For DEB-based systems (Debian GNU/Linux, Ubuntu) systems, drbd-utils and drbd-module-`uname -r` packages are installed by using apt install,

LINBIT provides a Docker registry for its commercial support customers. The registry is accessible through the host name ‘drbd.io’.

Before you can pull images, you have to log in to the registry:

After a successful login, you can pull images. To test your login and the registry, start by issuing the following command:

Several Linux distributions provide DRBD, including prebuilt binary packages. Support for these builds, if any, is being provided by the associated distribution vendor. Their release cycle may lag behind DRBD source releases.

Releases generated by Git tags on github are snapshots of the Git repository at the given time. You most likely do not want to use these. They might lack things such as generated man pages, the configure script, and other generated files. If you want to build from a tar file, use the ones provided by us.

All our projects contain standard build scripts (e.g., Makefile, configure). Maintaining specific information per distribution (e.g., documenting broken build macros) is too cumbersome, and historically the information provided in this section got outdated quickly. If you don’t know how to build software the standard way, please consider using packages provided by LINBIT.

The source tar files for both current and historic DRBD releases are available for download from https://pkg.linbit.com/. Source tar files, by convention, are named drbd-x.y.z.tar.gz, for example, drbd-utils-x.y.z.tar.gz, where x, y and z refer to the major, minor and bug fix release numbers.

DRBD’s compressed source archive is less than half a megabyte in size. After downloading a tar file, you can decompress its contents into your current working directory, by using the tar -xzf command.

For organizational purposes, decompress DRBD into a directory normally used for keeping source code, such as /usr/src or /usr/local/src. The examples in this guide assume /usr/src.

DRBD’s source code is kept in a public Git repository. You can browse this online at https://github.com/LINBIT. The DRBD software consists of these projects:

Source code can be obtained by either cloning Git repositories or downloading release tar files. There are two minor differences between an unpacked source tar file and a Git checkout of the same release:

After cloning the DRBD and related utilities source code repositories to your local host, you can proceed to building DRBD from the source code.

Provided your DRBD build completed successfully, you will be able to install DRBD by issuing the command:

The DRBD userspace management tools (drbdadm, drbdsetup, and drbdmeta) will now be installed in the prefix path that was passed to configure, typically /sbin/.

Note that any kernel upgrade will require you to rebuild and reinstall the DRBD kernel module to match the new kernel.

Some distributions allow to register kernel module source directories, so that rebuilds are done as necessary. See e.g. dkms(8) on Debian.

The DRBD userspace tools, in contrast, need only to be rebuilt and reinstalled when upgrading to a new DRBD version. If at any time you upgrade to a new kernel and new DRBD version, you will need to upgrade both components.

The DRBD build system contains a facility to build RPM packages directly out of the DRBD source tree. For building RPMs, Checking Build Prerequisites applies essentially in the same way as for building and installing with make, except that you also need the RPM build tools, of course.

Also, see Preparing the Kernel Source Tree if you are not building against a running kernel with precompiled headers available.

The build system offers two approaches for building RPMs. The simpler approach is to simply invoke the rpm target in the top-level Makefile:

This approach will auto-generate spec files from pre-defined templates, and then use those spec files to build binary RPM packages.

The make rpm approach generates several RPM packages:

The other, more flexible approach is to have configure generate the spec file, make any changes you deem necessary, and then use the rpmbuild command:

The RPMs will be created wherever your system RPM configuration (or your personal ~/.rpmmacros configuration) dictates.

After you have created these packages, you can install, upgrade, and uninstall them as you would any other RPM package in your system.

Note that any kernel upgrade will require you to generate a new kmod-drbd package to match the new kernel; see also Kernel Application Binary Interface warning for some distributions.

The DRBD userland packages, in contrast, need only be recreated when upgrading to a new DRBD version. If at any time you upgrade to a new kernel and new DRBD version, you will need to upgrade both packages.

The DRBD build system contains a facility to build Debian packages directly out of the DRBD source tree. For building Debian packages, Checking Build Prerequisites applies essentially in the same way as for building and installing with make, except that you of course also need the dpkg-dev package containing the Debian packaging tools, and fakeroot if you want to build DRBD as a non-root user (highly recommended). All DRBD sub-projects (kernel module and drbd-utils) support Debian package building.

Also, see Preparing the Kernel Source Tree if you are not building against a running kernel with precompiled headers available.

The DRBD source tree includes a debian subdirectory containing the required files for Debian packaging. That subdirectory, however, is not included in the DRBD source tar files — instead, you will need to create a Git checkout of a tag associated with a specific DRBD release.

Once you have created your checkout in this fashion, you can issue the following commands to build DRBD Debian packages:

This build process will create the following Debian packages:

After you have created these packages, you can install, upgrade, and uninstall them as you would any other Debian package in your system.

The drbd-utils packages supports Debian’s dpkg-reconfigure facility, which can be used to switch which versions of the man-pages are shown by default (8.3, 8.4, or 9.0).

Building and installing the actual kernel module from the installed module source package is easily accomplished via Debian’s module-assistant facility:

You can also use the shorthand form of the above command:

Note that any kernel upgrade will require you to rebuild the kernel module (with module-assistant, as just described) to match the new kernel. The drbd-utils and drbd-module-source packages, in contrast, only need to be recreated when upgrading to a new DRBD version. If at any time you upgrade to a new kernel and new DRBD version, you will need to upgrade both packages.

Starting from DRBD9, automatic updates of the DRBD kernel module are possible with the help of dkms(8). All that is needed is to install the drbd-dkms Debian package.

DRBD allows you to reconfigure resources while they are operational. To that end,

drbdadm adjust then hands off to drbdsetup to make the necessary adjustments to the configuration. As always, you are able to review the pending drbdsetup invocations by running drbdadm with the -d (dry-run) option.

Manually switching a resource’s role from secondary to primary (promotion) or vice versa (demotion) is done using one of the following commands:

In single-primary mode (DRBD’s default), any resource can be in the primary role on only one node at any given time while the connection state is Connected. Therefore, issuing drbdadm primary <resource> on one node while the specified resource is still in the primary role on another node will result in an error.

A resource configured to allow dual-primary mode can be switched to the primary role on two nodes; this is, for example, needed for online migration of virtual machines.

If not using a cluster manager and looking to handle failovers manually in a passive/active configuration, the process is as follows.

On the current primary node, stop any applications or services using the DRBD device, unmount the DRBD device, and demote the resource to secondary.

Now on the node you want to make primary promote the resource and mount the device.

If you’re using the auto-promote feature, you don’t need to change the roles (Primary/Secondary) manually; only stopping of the services and unmounting, respectively mounting, is necessary.

Included with drbd-utils versions since 9.26.0, there is a “graceful shutdown” service, drbd-graceful-shutdown.service. This service ensures that the DRBD “last man standing” behavior applies to your cluster when shutting down nodes.

The graceful shutdown service is started automatically by the udev service at the moment the first DRBD device is created. At system shutdown, the graceful shutdown service ensures that all nodes hosting DRBD resources shut down services in a proper sequence so that the last node keeps quorum.

During a normal system shutdown sequence without this intervention, networking often stops before a system unmounts file systems and takes down DRBD devices. Without the graceful shutdown service, to make it so that the last node to shut down keeps DRBD quorum, you would need to manually unmount file systems and down DRBD resources before shutting down nodes.

By virtue of the graceful shutdown service running on a node, when shutting down, a node marks its DRBD resources outdated, stops its running DRBD services, and then networking services are allowed to stop. This shutdown sequence makes it so that the DRBD peer nodes leaving the cluster are able to communicate their outdated status over the network to the last node, and in this way the last node to leave the cluster will keep quorum.

Upgrading DRBD is a fairly simple process. This section contains warnings or important information regarding upgrading to a particular DRBD 9 version from another DRBD 9 version.

If you are upgrading DRBD from 8.4.x to 9.x, refer to the instructions within the Appendix.

Dual-primary mode allows a resource to assume the primary role simultaneously on more than one node. Doing so is possible on either a permanent or a temporary basis.

Normally, one tries to ensure that background synchronization (which makes the data on the synchronization target temporarily inconsistent) completes as quickly as possible. However, it is also necessary to keep background synchronization from hogging all bandwidth otherwise available for foreground replication, which would be detrimental to application performance. Therefore, you must configure the synchronization bandwidth to match your hardware — which you may do in a permanent fashion or on-the-fly.

Likewise, and for the same reasons, it does not make sense to set a synchronization rate that is higher than the bandwidth available on the replication network.

Checksum-based synchronization is not enabled for resources by default. To enable it, add the following lines to your resource configuration in /etc/drbd.conf:

<algorithm> may be any message digest algorithm supported by the kernel crypto API in your system’s kernel configuration. Normally, you should be able to choose at least from sha1, md5, and crc32c.

If you make this change to an existing resource, as always, synchronize your drbd.conf to the peer, and run drbdadm adjust <resource> on both nodes.

In an environment where the replication bandwidth is highly variable (as would be typical in WAN replication setups), the replication link may occasionally become congested. In a default configuration, this would cause I/O on the primary node to block, which is sometimes undesirable.

Instead, you may configure DRBD to suspend the ongoing replication in this case, causing the Primary’s data set to pull ahead of the Secondary. In this mode, DRBD keeps the replication channel open — it never switches to disconnected mode — but does not actually replicate until sufficient bandwidth becomes available again.

The following example is for a DRBD Proxy configuration:

It is usually wise to set both congestion-fill and congestion-extents together with the pull-ahead option.

A good value for congestion-fill is 90%

A good value for congestion-extents is 90% of your configured al-extents for the affected resources.

DRBD’s strategy for handling lower-level I/O errors is determined by the on-io-error option, included in the resource disk configuration in /etc/drbd.conf:

You may, of course, set this in the common section too, if you want to define a global I/O error handling policy for all resources.

The on-io-error option is independent from the on-no-data-accessible option. Some on-io-error strategies involve retrying I/O requests on peer disks. The on-no-data-accessible option setting dictates DRBD behavior when I/O requests are unsuccessful on all disks.

You can set the on-io-error <strategy> to one of the following:

This is the default and recommended option. On the occurrence of a lower-level I/O error, the node drops its backing device, and continues in diskless mode.

This causes DRBD to change the disk status to Inconsistent, mark the failed block as inconsistent in the DRBD quick-sync bitmap, and retry the I/O operation on a peer (secondary node) disk. If the I/O operation succeeds on at least one peer, then a write operation is considered successful. DRBD will retry a read operation until there are no more peer disks to try to read from.

Invokes the command defined as the local I/O error handler. This requires that a corresponding local-io-error command invocation is defined in the resource’s handlers section. It is entirely left to the administrator’s discretion to implement I/O error handling using the command (or script) invoked by local-io-error.

You may reconfigure a running resource’s I/O error handling strategy by following this process:

Replication traffic integrity checking is not enabled for resources by default. To enable it, add the following lines to your resource configuration in /etc/drbd.conf:

<algorithm> may be any message digest algorithm supported by the kernel crypto API in your system’s kernel configuration. Normally, you should be able to choose at least from sha1, md5, and crc32c.

If you make this change to an existing resource, as always, synchronize your drbd.conf to the peer, and run drbdadm adjust <resource> on both nodes.

When growing (extending) DRBD volumes, you need to grow from bottom to top. First, you need to extend the backing block devices on all nodes. Then you can tell DRBD to use the new space.

Once the DRBD volume is extended, you still need to propagate that change into whatever upper layers might be using the DRBD volume, for example, by extending the file system, or making a VM running with this volume attached aware of the new “disk size”.

Doing this typically means taking the following steps:

Note that different file systems have different capabilities and different sets of management tools. For example XFS can only grow. You point its tool to the active mount point: xfs_growfs /where/you/have/it/mounted.

While the EXT family can both grow (even online), and also shrink (only offline; you have to unmount it first). To resize an ext3 or ext4, you would point the tool not to the mount point, but to the (mounted) block device: resize2fs /dev/drbd#

Obviously use the correct DRBD (as displayed by mount or df -T, while mounted), and not the backing block device. If DRBD is up, that’s not supposed to work anyways (resize2fs: Device or resource busy while trying to open /dev/mapper/VG-LV Couldn’t find valid filesystem superblock.). If you tried to do that offline (with DRBD stopped), you may corrupt DRBD metadata if you ran the file system tools directly against the backing LV or partition. So don’t.

You do the file system resize only once on the Primary, against the active DRBD device. DRBD replicates the changes to the file system structure. That is what you have it for.

Also, don’t use resize2fs on XFS volumes, or XFS tools on EXT, or …​ but the right tool for the file system in use.

resize2fs: Bad magic number in super-block while trying to open /dev/drbd7 is probably just trying to tell you that this is not an EXT file system, and you should try an other tool instead. Maybe xfs_growfs? But as mentioned, that does not take the block device, but the mount point as argument.

When shrinking (reducing) DRBD volumes, you need to shrink from top to bottom. So first verify that no one is using the space you want to cut off. Next, shrink the file system (if your file system supports that). Then tell DRBD to stop using that space, which is not so easy with DRBD internal metadata, because they are by design “at the end” of the backing device.

Once you are sure that DRBD won’t use the space anymore either, you can cut it off from the backing device, for example using lvreduce.

See also Shrinking Online, Shrinking Offline.

Disabling DRBD’s flushes when running without BBWC, or on BBWC with a depleted battery, is likely to cause data loss and should not be attempted.

DRBD allows you to enable and disable backing device flushes separately for the replicated data set and DRBD’s own metadata. Both of these options are enabled by default. If you want to disable either (or both), you would set this in the disk section for the DRBD configuration file, /etc/drbd.conf.

To disable disk flushes for the replicated data set, include the following line in your configuration:

To disable disk flushes on DRBD’s metadata, include the following line:

After you have modified your resource configuration (and synchronized your /etc/drbd.conf between nodes, of course), you may enable these settings by issuing this command on both nodes:

If only one of the servers has a BBWC^[7], you should move the setting into a host section, like this:

A three-node setup involves one DRBD device stacked atop another.

A node might be permanently diskless in DRBD. Here is a configuration example showing a resource with 3 diskful nodes (servers) and one permanently diskless node (client).

For permanently diskless nodes no bitmap slot gets allocated. For such nodes the diskless status is displayed in green color since it is not an error or unexpected state. See The Client Mode for internal details.

Given the (example) policy that data needs to be available on 3 nodes, you need at least 3 servers for your setup.

Now, as your storage demands grow, you will encounter the need for additional servers. Rather than having to buy 3 more servers at the same time, you can rebalance your data across a single additional node.

In the figure above you can see the before and after states: from 3 nodes with three 25TiB volumes each (for a net 75TiB), to 4 nodes, with net 100TiB.

To redistribute the data across your cluster you have to choose a new node, and one where you want to remove this DRBD resource.
Please note that removing the resource from a currently active node (that is, where DRBD is Primary) will involve either migrating the service or running this resource on this node as a DRBD client; it’s easier to choose a node in Secondary role. (Of course, that might not always be possible.)

This section describes how you can configure the DRBD quorum feature to avoid split-brain situations and data divergence in your high-availability clusters.

You enable quorum for a DRBD resource by setting the quorum option within the options section of a DRBD resource configuration file to either majority, all, or a numerical value. By default, DRBD quorum is not enabled and the option is set to off.

An example DRBD resource configuration that enables DRBD quorum is as follows:

You can also enable quorum globally, by setting the quorum option within the options subsection of the common section within the global DRBD configuration file, /etc/drbd.d/global_common.conf. Enabling quorum and any other related options within the global configuration file will affect all DRBD resources, unless the same option is set within a specific DRBD resource configuration file. In that case, the option value set within a DRBD resource configuration file will take precedence.

An example global configuration that enables DRBD quorum is as follows:

For the unlikely case that you want to remove DRBD, here are the necessary steps.

The DRBD Proxy processes can either be located directly on the machines where DRBD is set up, or they can be placed on distinct dedicated servers. A DRBD Proxy instance can serve as a proxy for multiple DRBD devices distributed across multiple nodes.

DRBD Proxy is completely transparent to DRBD. Typically you will expect a high number of data packets in flight, therefore the activity log should be reasonably large. Since this may cause longer re-sync runs after the failure of a primary node, it is recommended to enable the DRBD csums-alg setting.

For more information about the rationale for the DRBD Proxy, please see the feature explanation Long-distance Replication through DRBD Proxy.

The DRBD Proxy 3 uses several kernel features that are only available since 2.6.26, so running it on older systems (for example, RHEL 5) is not possible. Here we can still provide DRBD Proxy 1 packages, though^[9].

To obtain DRBD Proxy, please contact your LINBIT sales representative. Unless instructed otherwise, please always use the most recent DRBD Proxy release.

To install DRBD Proxy on Debian and Debian-based systems, use the dpkg tool as follows (replace version with your DRBD Proxy version, and architecture with your target architecture):

To install DRBD Proxy on RPM based systems (like SLES or RHEL) use the RPM tool as follows (replace version with your DRBD Proxy version, and architecture with your target architecture):

Also install the DRBD administration program drbdadm since it is required to configure DRBD Proxy.

This will install the DRBD Proxy binaries as well as an init script which usually goes into /etc/init.d. Please always use the init script to start/stop DRBD proxy since it also configures DRBD Proxy using the drbdadm tool.

When obtaining a license from LINBIT, you will be sent a DRBD Proxy license file which is required to run DRBD Proxy. The file is called drbd-proxy.license, it must be copied into the /etc directory of the target machines, and be owned by the user/group drbdpxy.

DRBD Proxy can be configured using LINSTOR as described in the LINSTOR User’s Guide.

DRBD Proxy can also be configured by editing resource files. It is configured by an additional section called proxy and additional proxy on sections within the host sections.

Below is a DRBD configuration example for proxies running directly on the DRBD nodes:

The inside IP address is used for communication between DRBD and the DRBD Proxy, whereas the outside IP address is used for communication between the proxies. The latter channel might have to be allowed in your firewall setup.

drbdadm offers the proxy-up and proxy-down subcommands to configure or delete the connection to the local DRBD Proxy process of the named DRBD resource(s). These commands are used by the start and stop actions which /etc/init.d/drbdproxy implements.

The DRBD Proxy has a low level configuration tool, called drbd-proxy-ctl. When called without any option it operates in interactive mode.

To pass a command directly, avoiding interactive mode, use the -c parameter followed by the command.

To display the available commands use:

Note the double quotes around the command being passed.

Here is a list of commands; while the first few ones are typically only used indirectly (via drbdadm proxy-up resp. drbdadm proxy-down), the latter ones give various status information.

Since DRBD Proxy version 3 the proxy allows to enable a few specific plugins for the WAN connection. The currently available plugins are zstd, lz4, zlib, and lzma (all software compression).

zstd (Zstandard) is a real-time compression algorithm, providing high compression ratios. It offers a very wide range of compression / speed trade-off, while being backed by a very fast decoder. Compression rates are dependent on “level” parameter which can be arranged between 1 to 22. Over level 20, DRBD Proxy will require more memory.

lz4 is a very fast compression algorithm; the data typically gets compressed down by 1:2 to 1:4, half- to two-thirds of the bandwidth can be saved.

The zlib plugin uses the GZIP algorithm for compression; it uses a bit more CPU than lz4, but gives a ratio of 1:3 to 1:5.

The lzma plugin uses the liblzma2 library. It can use dictionaries of several hundred MiB; these allow for very efficient delta-compression of repeated data, even for small changes. lzma needs much more CPU and memory, but results in much better compression than zlib — real-world tests with a VM sitting on top of DRBD gave ratios of 1:10 to 1:40. The lzma plugin has to be enabled in your license.

Contact LINBIT to find the best settings for your environment – it depends on the CPU (speed, number of threads), available memory, input and available output bandwidth, and expected I/O spikes. Having a week of sysstat data already available helps in determining the configuration, too.

DRBD Proxy logs events through syslog using the LOG_DAEMON facility. Usually you will find DRBD Proxy events in /var/log/daemon.log.

Enabling debug mode in DRBD Proxy can be done with the following command.

For example, if proxy fails to connect it will log something like Rejecting connection because I can’t connect on the other side. In that case, please check if DRBD is running (not in StandAlone mode) on both nodes and if both proxies are running. Also double-check your configuration.

DRBD and the DRBD administrative tool, drbdadm, return POSIX error codes. If you need to get more information about a specific error code number, you can use the following command, provided that Perl is installed in your environment. For example, to get information about error code number 11, enter:

How to deal with hard disk failure depends on the way DRBD is configured to handle disk I/O errors (see Disk Error Handling Strategies), and on the type of metadata configured (see DRBD Metadata).

When DRBD detects that its peer node is down (either by true hardware failure or manual intervention), DRBD changes its connection state from Connected to Connecting and waits for the peer node to reappear. The DRBD resource is then said to operate in disconnected mode. In disconnected mode, the resource and its associated block device are fully usable, and may be promoted and demoted as necessary, but no block modifications are being replicated to the peer node. Instead, DRBD stores which blocks are being modified while disconnected, on a per-peer basis.

DRBD detects split brain at the time connectivity becomes available again and the peer nodes exchange the initial DRBD protocol handshake. If DRBD detects that both nodes are (or were at some point, while disconnected) in the primary role, it immediately tears down the replication connection. The tell-tale sign of this is a message like the following appearing in the system log:

After split brain has been detected, one node will always have the resource in a StandAlone connection state. The other might either also be in the StandAlone state (if both nodes detected the split brain simultaneously), or in Connecting (if the peer tore down the connection before the other node had a chance to detect split brain).

At this point, unless you configured DRBD to automatically recover from split brain, you must manually intervene by selecting one node whose modifications will be discarded (this node is referred to as the split brain victim). This intervention is made with the following commands:

On the other node (the split brain survivor), if its connection state is also StandAlone, you would enter:

You may omit this step if the node is already in the Connecting state; it will then reconnect automatically.

Upon connection, your split brain victim immediately changes its connection state to SyncTarget, and gets its modifications overwritten by the other node(s).

After re-synchronization has completed, the split brain is considered resolved and the nodes form a fully consistent, redundant replicated storage system again.

The DRBD administration tool, drbdadm, includes a force secondary option, secondary --force. If DRBD quorum was configured to suspend DRBD resource I/O operations upon loss of quorum, the force secondary option will allow you to gracefully recover the node that lost quorum and reintegrate it with the other nodes.

Requirements:

You can use the command drbdadm secondary --force <all|resource_name> to demote a primary node to secondary, in cases where you are trying to recover a primary node that lost quorum. The argument to this command can be either a single DRBD resource name or all to demote the node to a secondary role for all its DRBD resources.

By using this command on the primary node that lost quorum with suspended I/O operations, all the suspended I/O requests and newly submitted I/O requests will terminate with I/O errors. You can then usually unmount the file system and reconnect the node to the other nodes in your cluster. An edge case is a file system opener that does not do any I/O and just idles around. Such processes need to be removed manually before unmounting will succeed or with the help of external tools such as fuser -k, or the OCF file system resource agent in clustered setups.

Along with the DRBD administration tool’s force secondary option, you can also add the on-suspended-primary-outdated option to a DRBD resource configuration file and set it to the keyword value force-secondary. You will also need to add the resource role conflict (rr-conflict) option to the DRBD resource configuration file’s net section, and set it to retry-connect. This enables DRBD to automatically recover a primary node that loses quorum with suspended I/O operations. With these options configured, when such a node connects to a cluster partition that has a more recent data set, DRBD automatically demotes the primary node that lost quorum and has suspended I/O operations. Additional configurations, for example in a handlers section of the resource configuration file, as well as additional configurations within a cluster manager, may also be necessary to complete a fully automatic recovery setup.

Settings within a DRBD resource configuration file’s options section that cover this scenario could look like this:

DRBD Reactor can be installed from source files found within the project’s GitHub repository. See the instructions there for details and any prerequisites.

Alternatively, LINBIT customers can install DRBD Reactor from prebuilt packages, available from LINBIT’s drbd-9 packages repository.

Once installed, you can verify DRBD Reactor’s version number by using the drbd-reactor --version command.

Because DRBD Reactor has many different uses, it was split into two components: a core component and a plugin component.

Before you can run DRBD Reactor, you must configure it. Global configurations are made within a main TOML configuration file, which should be created here: /etc/drbd-reactor.toml. The file has to be a valid TOML (https://toml.io) file. Plugin configurations should be made within snippet files that can be placed into the default DRBD Reactor snippets directory, /etc/drbd-reactor.d, or into another directory if specified in the main configuration file. An example configuration file can be found in the example directory of the DRBD Reactor GitHub repository.

For documentation purposes only, the example configuration file mentioned above contains example plugin configurations. However, for deployment, plugin configurations should always be made within snippet files.

You can use the DRBD Reactor CLI utility, drbd-reactorctl, to control the DRBD Reactor daemon and its plugins.

With the drbd-reactorctl utility, you can:

For greater control of some of the above actions, there are additional options available. The drbd-reactorctl man page has more details and syntax information.

In this example, you will use DRBD Reactor and the promoter plugin to create a highly available file system mount within a cluster.

Prerequisites:

The DRBD resource, ha-mount, should have the following settings configured in its DRBD resource configuration file:

First, make one of your nodes Primary for the ha-mount resource.

Then create a file system on the DRBD backed device. The ext4 file system is used in this example.

Make the node Secondary because after further configurations, DRBD Reactor and the Promoter plugin will control promoting nodes.

On all nodes that should be able to mount the DRBD backed device, create a systemd unit file:

Next, on the same nodes as the previous step, create a configuration file for the DRBD Reactor promoter plugin:

To apply the configuration, enable and start the DRBD Reactor service on all nodes. If the DRBD Reactor service is already running, reload it instead.

Next, verify which cluster node is in the Primary role for the ha-mount resource and has the backing device mounted.

Test a simple failover situation on the Primary node by using the DRBD Reactor CLI utility to disable the ha-mount configuration.

Run the DRBD Reactor status command again to verify that another node is now in the Primary role and has the file system mounted.

After testing failover, you can enable the configuration on the node you disabled it on earlier.

As a next step, you may want to read the LINSTOR User’s Guide section on creating a highly available LINSTOR cluster. There, DRBD Reactor is used to manage the LINSTOR Controller as a service so that it is highly available within your cluster.

DRBD Reactor’s Prometheus monitoring plugin acts as a Prometheus compatible endpoint for DRBD resources and exposes various DRBD metrics. You can find a list of the available metrics in the documentation folder in the project’s GitHub repository.

Prerequisites:

To enable the Prometheus plugin, create a simple configuration file snippet on all DRBD Reactor nodes that you are monitoring.

Reload the DRBD Reactor service on all nodes that you are monitoring.

Add the following DRBD Reactor monitoring endpoint to your Prometheus configuration file’s scrape_configs section. Replace “node-x” in the targets lines below with either hostnames or IP addresses for your DRBD Reactor monitoring endpoint nodes. Hostnames must be resolvable from your Prometheus monitoring node.

Then, assuming it is already enabled and running, reload the Prometheus service by entering sudo systemctl reload prometheus.service.

Next, you can open your Grafana server’s URL with a web browser. If the Grafana server service is running on the same node as your Prometheus monitoring service, the URL would look like: http://<node_IP_address_or_hostname>:3000.

You can then log into the Grafana server web UI, add a Prometheus data source, and then add or import a Grafana dashboard that uses your Prometheus data source. An example dashboard is available at the Grafana Labs dashboards marketplace. An example dashboard is also available as a downloadable JSON file here, at the DRBD Reactor GitHub project site.

Pacemaker is a sophisticated, feature-rich, and widely deployed cluster resource manager for the Linux platform. It includes a rich set of documentation. To understand this chapter, reading the following documents is highly recommended:

In this section you will see that using autonomous DRBD storage can look like local storage; so integrating in a Pacemaker cluster is done by pointing your mount points at DRBD.

First of all, we will use the auto-promote feature of DRBD, so that DRBD automatically sets itself Primary when needed. This will probably apply to all of your resources, so setting that as a default “yes” in the common section makes sense:

Now you just need to use your storage, for example, through a filesystem:

Essentially all that is needed is a mountpoint (/var/lib/mysql in this example) where the DRBD resource gets mounted.

Provided that Pacemaker has control, it will only allow a single instance of that mount across your cluster.

See also Importing DRBD’s Promotion Scores into the CIB for additional information about ordering constraints for system startup and more.

This section explains how to enable a DRBD-backed service in a Pacemaker cluster.

The ocf:linbit:drbd OCF resource agent provides Master/Slave capability, allowing Pacemaker to start and monitor the DRBD resource on multiple nodes and promoting and demoting as needed. You must, however, understand that the DRBD RA disconnects and detaches all DRBD resources it manages on Pacemaker shutdown, and also upon enabling standby mode for a node.

To enable a DRBD-backed configuration for a MySQL database in a Pacemaker CRM cluster with the drbd OCF resource agent, you must create both the necessary resources, and Pacemaker constraints to ensure your service only starts on a previously promoted DRBD resource. You may do so using the crm shell, as outlined in the following example:

After this, your configuration should be enabled. Pacemaker now selects a node on which it promotes the DRBD resource, and then starts the DRBD-backed resource group on that same node.

See also Importing DRBD’s Promotion Scores into the CIB for additional information about location constraints for placing the Master role.

This section outlines the steps necessary to prevent Pacemaker from promoting a DRBD Master/Slave resource when its DRBD replication link has been interrupted. This keeps Pacemaker from starting a service with outdated data and causing an unwanted “time warp” in the process.

To enable any resource-level fencing for DRBD, you must add the following lines to your resource configuration:

You will also have to make changes to the handlers section depending on the cluster infrastructure being used.

Corosync-based Pacemaker clusters can use the functionality explained in Resource-level Fencing Using the Cluster Information Base (CIB).

Stacked resources allow DRBD to be used for multi-level redundancy in multiple-node clusters, or to establish off-site disaster recovery capability. This section describes how to configure DRBD and Pacemaker in such configurations.

This is a somewhat advanced setup usually employed in split-site configurations. It involves two separate Pacemaker clusters, where each cluster has access to a separate Storage Area Network (SAN). DRBD is then used to replicate data stored on that SAN, across an IP link between sites.

Consider the following illustration to describe the concept.

Which of the individual nodes in each site currently acts as the DRBD peer is not explicitly defined — the DRBD peers are said to float; that is, DRBD binds to virtual IP addresses not tied to a specific physical machine.

Since this type of setup deals with shared storage, configuring and testing STONITH is absolutely vital for it to work properly.

Every DRBD resource exposes a promotion score on each node where it is configured. It is a numeric value that might be 0 or positive. The value reflects how desirable it is to promote the resource to master on this particular node. A node that has an UpToDate disk and two UpToDate replicas has a higher score than a node with an UpToDate disk and just one UpToDate replica.

During startup, the promotion score is 0. E.g., before the DRBD device has its backing device attached, or, if quorum is enabled, before quorum is gained. A value of 0 indicates that a promotion request will fail, and is mapped to a pacemaker score that indicates must not run here.

The drbd-attr OCF resource agent imports these promotion scores into node attributes of a Pacemaker cluster. It needs to be configured like this:

These are transient attributes (have a lifetime of reboot in pacemaker speak). That means, after a reboot of the node, or local restart of pacemaker, those attributes will not exist until an instance of drbd-attr is started on that node.

You can inspect the generated attributes with crm_mon -A -1.

These attributed can be used in constraints for services that depend on the DRBD devices, or, when managing DRBD with the ocf:linbit:drbd resource agent, for the Master role of that DRBD instance.

Here is an example location constraint for the example resource from Using DRBD as a Background Service in a Pacemaker Cluster

This means, provided that the attribute is not defined, the fs_mysql file system cannot be mounted here. When the attribute is defined, its value becomes the score of the location constraint.

This can also be used to cause Pacemaker to migrate a service away when DRBD loses a local backing device. Because a failed backing block device causes the promotion score to drop, other nodes with working backing devices will expose higher promotion scores.

The attributes are updated live, independent of the resource-agent’s monitor operation, with a dampening delay of 5 seconds by default.

The resource agent has these optional parameters, see also its man page ocf_linbit_drbd-attr(7):

LVM2 is an implementation of logical volume management in the context of the Linux device mapper framework. It has practically nothing in common, other than the name and acronym, with the original LVM implementation. The old implementation (now retroactively named “LVM1”) is considered obsolete; it is not covered in this section.

When working with LVM, it is important to understand its most basic concepts:

A PV is an underlying block device exclusively managed by LVM. PVs can either be entire hard disks or individual partitions. It is common practice to create a partition table on the hard disk where one partition is dedicated to the use by the Linux LVM.

A VG is the basic administrative unit of the LVM. A VG may include one or more several PVs. Every VG has a unique name. A VG may be extended during runtime by adding additional PVs, or by enlarging an existing PV.

LVs may be created during runtime within VGs and are available to the other parts of the kernel as regular block devices. As such, they may be used to hold a file system, or for any other purpose block devices may be used for. LVs may be resized while they are online, and they may also be moved from one PV to another (provided that the PVs are part of the same VG).

Snapshots are temporary point-in-time copies of LVs. Creating snapshots is an operation that completes almost instantly, even if the original LV (the origin volume) has a size of several hundred GiByte. Usually, a snapshot requires significantly less space than the original LV.

Because an existing LVM logical volume is simply a block device in Linux terms, you can use one as a DRBD backing device. To use an LVM logical volume in this way, you simply create one, and then reference it in a DRBD resource configuration just as you would a physical backing disk.

The following example assumes that an LVM volume group named foo already exists on two nodes of in your LVM-enabled cluster, and that you want to create a DRBD resource named r0 using a logical volume in that volume group.

First, you create the logical volume on both nodes in your cluster:

After this, you will have a block device named /dev/foo/bar on both nodes.

Then, you can simply enter the newly created volumes in your resource configuration:

Now you can continue to bring your resource up, just as you would if you were using non-LVM block devices.

While DRBD is synchronizing, the SyncTarget‘s state is Inconsistent until the synchronization completes. If in this situation the SyncSource happens to fail (beyond repair), this puts you in an unfortunate position: the node with good data is dead, and the surviving node has bad (inconsistent) data.

When serving DRBD off an LVM Logical Volume, you can mitigate this problem by creating an automated snapshot when synchronization starts, and automatically removing that same snapshot once synchronization has completed successfully.

To enable automated snapshotting during resynchronization, add the following lines to your resource configuration:

The two scripts parse the $DRBD_RESOURCE environment variable which DRBD automatically passes to any handler it invokes. The snapshot-resync-target-lvm.sh script then creates an LVM snapshot for any volume the resource contains, then synchronization kicks off. In case the script fails, the synchronization does not commence.

Once synchronization completes, the unsnapshot-resync-target-lvm.sh script removes the snapshot, which is then no longer needed. In case unsnapshotting fails, the snapshot continues to linger around.

If at any time your SyncSource does fail beyond repair and you decide to revert to your latest snapshot on the peer, you may do so by issuing the lvconvert -M command.

To prepare a DRBD resource for use as a Physical Volume, it is necessary to create a PV signature on the DRBD device. To do this, issue one of the following commands on the node where the resource is currently in the primary role:

or

Now, it is necessary to include this device in the list of devices LVM scans for PV signatures. To do this, you must edit the LVM configuration file, normally named /etc/lvm/lvm.conf. Find the line in the devices section that contains the filter keyword and edit it accordingly. If all your PVs are to be stored on DRBD devices, the following is an appropriate filter option:

This filter expression accepts PV signatures found on any DRBD devices, while rejecting (ignoring) all others.

If you want to use stacked resources as LVM PVs, then you will need a more explicit filter configuration. You need to verify that LVM detects PV signatures on stacked resources, while ignoring them on the corresponding lower-level resources and backing devices. This example assumes that your lower-level DRBD resources use device minors 0 through 9, whereas your stacked resources are using device minors from 10 upwards:

This filter expression accepts PV signatures found only on the DRBD devices /dev/drbd10 through /dev/drbd19, while rejecting (ignoring) all others.

After modifying the lvm.conf file, you must run the vgscan command so LVM discards its configuration cache and re-scans devices for PV signatures.

You may of course use a different filter configuration to match your particular system configuration. What is important to remember, however, is that you need to:

In addition, you should disable the LVM cache by setting:

After disabling the LVM cache, remove any stale cache entries by deleting /etc/lvm/cache/.cache.

You must repeat the above steps on the peer nodes, too.

When you have configured your new PV, you may proceed to add it to a Volume Group, or create a new Volume Group from it. The DRBD resource must, of course, be in the primary role while doing so.

When you have created your VG, you may start carving Logical Volumes out of it, using the lvcreate command (as with a non-DRBD-backed Volume Group).

Occasionally, you may want to add new DRBD-backed Physical Volumes to a Volume Group. Whenever you do so, a new volume should be added to an existing resource configuration. This preserves the replication stream and ensures write fidelity across all PVs in the VG.

Extend your resource configuration to include an additional volume, as in the following example:

Verify that your DRBD configuration is identical across nodes, then issue:

This will implicitly call drbdsetup new-minor r0 1 to enable the new volume 1 in the resource r0. Once the new volume has been added to the replication stream, you may initialize and add it to the volume group:

This will add the new PV /dev/drbd/by-res/<resource>/1 to the <name> VG, preserving write fidelity across the entire VG.

The process of transferring volume groups between peers and making the corresponding logical volumes available can be automated. The Pacemaker LVM resource agent is designed for exactly that purpose.

To put an existing, DRBD-backed volume group under Pacemaker management, run the following commands in the crm shell:

After you have committed this configuration, Pacemaker will automatically make the r0 volume group available on whichever node currently has the Primary (Master) role for the DRBD resource.

The typical high availability use case for DRBD is to use a cluster resource manager (CRM) to handle the promoting and demoting of resources, such as DRBD replicated storage volumes. However, it is possible to use DRBD without a CRM.

You might want to do this in a situation when you know that you always want a particular node to promote a DRBD resource and you know that the peer nodes are never going to take over but are only being replicated to for disaster recovery purposes.

In this case, you can use a couple of systemd unit files to handle DRBD resource promotion and make sure that back-end LVM logical volumes are activated first. You also need to make the DRBD systemd unit file for your DRBD resource a dependency of whatever file system mount might be using the DRBD resource as a backing device.

To set this up, for example, given a hypothetical DRBD resource named webdata and a file system mount point of /var/lib/www, you might enter the following commands:

In this example, the X in drbdX is the volume number of your DRBD backing device for the webdata resource.

The drbd-wait-promotable@<DRBD-resource-name>.service is a systemd unit file that is used to wait for DRBD to connect to its peers and establish access to good data, before DRBD promotes the resource on the node.

The Red Hat Global File System (GFS) is Red Hat’s implementation of a concurrent-access shared storage file system. As any such filesystem, GFS allows multiple nodes to access the same storage device, in read/write fashion, simultaneously without risking data corruption. It does so by using a Distributed Lock Manager (DLM) which manages concurrent access from cluster members.

GFS was designed, from the outset, for use with conventional shared storage devices. Regardless, it is perfectly possible to use DRBD, in dual-primary mode, as a replicated storage device for GFS. Applications may benefit from reduced read/write latency due to the fact that DRBD normally reads from and writes to local storage, as opposed to the SAN devices GFS is normally configured to run from. Also, of course, DRBD adds an additional physical copy to every GFS filesystem, therefore adding redundancy to the concept.

GFS makes use of a cluster-aware variant of LVM, termed Cluster Logical Volume Manager or CLVM. As such, some parallelism exists between using DRBD as the data storage for GFS, and using DRBD as a Physical Volume for conventional LVM.

GFS file systems are usually tightly integrated with Red Hat’s own cluster management framework, the Red Hat Cluster. This chapter explains the use of DRBD in conjunction with GFS in the Red Hat Cluster context.

GFS, CLVM, and Red Hat Cluster are available in Red Hat Enterprise Linux (RHEL) and distributions derived from it, such as CentOS. Packages built from the same sources are also available in Debian GNU/Linux. This chapter assumes running GFS on a Red Hat Enterprise Linux system.

Since GFS is a shared cluster file system expecting concurrent read/write storage access from all cluster nodes, any DRBD resource to be used for storing a GFS filesystem must be configured in dual-primary mode. Also, it is recommended to use some of DRBD’s features for automatic recovery from split brain. To do all this, include the following lines in the resource configuration:

Once you have added these options to your freshly-configured resource, you may initialize your resource as you normally would. Since the allow-two-primaries option is set to yes for this resource, you will be able to promote the resourceto the primary role on two nodes.

GFS uses CLVM, the cluster-aware version of LVM, to manage block devices to be used by GFS. To use CLVM with DRBD, ensure that your LVM configuration

After you have created your new DRBD resource and completed your initial cluster configuration, you must enable and start the following system services on both nodes of your GFS cluster:

To create a GFS filesystem on your dual-primary DRBD resource, you must first initialize it as a Logical Volume for LVM.

Contrary to conventional, non-cluster-aware LVM configurations, the following steps must be completed on only one node due to the cluster-aware nature of CLVM:

CLVM will immediately notify the peer node of these changes; issuing lvs (or lvdisplay) on the peer node will list the newly created logical volume.

Now, you may proceed by creating the actual filesystem:

Or, for a GFS2 filesystem:

The -j option in this command refers to the number of journals to keep for GFS. This must be identical to the number of nodes with concurrent Primary role in the GFS cluster; since DRBD does not support more than two Primary nodes the value to set here is always 2.

The -t option, applicable only for GFS2 filesystems, defines the lock table name. This follows the format <cluster>:<name>, where <cluster> must match your cluster name as defined in /etc/cluster/cluster.conf. Therefore, only members of that cluster will be permitted to use the filesystem. By contrast, <name> is an arbitrary file system name unique in the cluster.

After you have created your filesystem, you may add it to /etc/fstab:

For a GFS2 filesystem, simply change the filesystem type:

Do not forget to make this change on both cluster nodes.

After this, you may mount your new filesystem by starting the gfs service (on both nodes):

From then on, if you have DRBD configured to start automatically on system startup, before the Pacemaker services and the gfs service, you will be able to use this GFS file system as you would use one that is configured on traditional shared storage.

The Oracle Cluster File System, version 2 (OCFS2) is a concurrent access shared storage file system developed by Oracle Corporation. Unlike its predecessor OCFS, which was specifically designed and only suitable for Oracle database payloads, OCFS2 is a general-purpose filesystem that implements most POSIX semantics. The most common use case for OCFS2 is arguably Oracle Real Application Cluster (RAC), but OCFS2 may also be used for load-balanced NFS clusters, for example.

Although originally designed for use with conventional shared storage devices, OCFS2 is equally well suited to be deployed on dual-Primary DRBD. Applications reading from the filesystem may benefit from reduced read latency due to the fact that DRBD reads from and writes to local storage, as opposed to the SAN devices OCFS2 otherwise normally runs on. In addition, DRBD adds redundancy to OCFS2 by adding an additional copy to every filesystem image, as opposed to just a single filesystem image that is merely shared.

Like other shared cluster file systems such as GFS, OCFS2 allows multiple nodes to access the same storage device, in read/write mode, simultaneously without risking data corruption. It does so by using a Distributed Lock Manager (DLM) which manages concurrent access from cluster nodes. The DLM itself uses a virtual file system (ocfs2_dlmfs) which is separate from the actual OCFS2 file systems present on the system.

OCFS2 may either use an intrinsic cluster communication layer to manage cluster membership and filesystem mount and unmount operation, or alternatively defer those tasks to the Pacemakercluster infrastructure.

OCFS2 is available in SUSE Linux Enterprise Server (where it is the primarily supported shared cluster file system), CentOS, Debian GNU/Linux, and Ubuntu Server Edition. Oracle also provides packages for Red Hat Enterprise Linux (RHEL). This chapter assumes running OCFS2 on a SUSE Linux Enterprise Server system.

Since OCFS2 is a shared cluster file system expecting concurrent read/write storage access from all cluster nodes, any DRBD resource to be used for storing a OCFS2 filesystem must be configured in dual-primary mode. Also, it is recommended to use some of DRBD’s features for automatic recovery from split brain. To do all this, include the following lines in the resource configuration:

It is not recommended to set the allow-two-primaries option to yes upon initial configuration. You should do so after the initial resource synchronization has completed.

Once you have added these options to your freshly-configured resource, you may initialize your resource as you normally would. After you set the allow-two-primaries option to yes for this resource, you will be able to promote the resourceto the primary role on both nodes.

Now, use OCFS2’s mkfs implementation to create the file system:

This will create an OCFS2 file system with two node slots on /dev/drbd0, and set the filesystem label to ocfs2_drbd0. You may specify other options on mkfs invocation; please see the mkfs.ocfs2 system manual page for details.

Xen is a virtualization framework originally developed at the University of Cambridge (UK), and later being maintained by XenSource, Inc. (now a part of Citrix). It is included in reasonably recent releases of most Linux distributions, such as Debian GNU/Linux (since version 4.0), SUSE Linux Enterprise Server (since release 10), Red Hat Enterprise Linux (since release 5), and many others.

Xen uses paravirtualization — a virtualization method involving a high degree of cooperation between the virtualization host and guest virtual machines — with selected guest operating systems for improved performance in comparison to conventional virtualization solutions (which are typically based on hardware emulation). Xen also supports full hardware emulation on CPUs that support the appropriate virtualization extensions; in Xen parlance, this is known as HVM ( “hardware-assisted virtual machine”).

Xen supports live migration, which refers to the capability of transferring a running guest operating system from one physical host to another, without interruption.

When a DRBD resource is used as a replicated Virtual Block Device (VBD) for Xen, it serves to make the entire contents of a DomU’s virtual disk available on two servers, which can then be configured for automatic failover. That way, DRBD does not only provide redundancy for Linux servers (as in non-virtual DRBD deployment scenarios), but also for any other operating system that can run virtually under Xen — which, in essence, includes any operating system available on 32- or 64-bit Intel compatible architectures.

For Xen Domain-0 kernels, it is recommended to load the DRBD module with the parameter disable_sendpage set to 1. To do so, create (or open) the file /etc/modprobe.d/drbd.conf and enter the following line:

Configuring a DRBD resource that is to be used as a Virtual Block Device (VBD) for Xen is fairly straightforward — in essence, the typical configuration matches that of a DRBD resource being used for any other purpose. However, if you want to enable live migration for your guest instance, you need to enable dual-primary modefor this resource:

Enabling dual-primary mode is necessary because Xen, before initiating live migration, checks for write access on all VBDs a resource is configured to use on both the source and the destination host for the migration.

To use a DRBD resource as the virtual block device, you must add a line like the following to your Xen DomU configuration:

This example configuration makes the DRBD resource named resource available to the DomU as /dev/xvda in read/write mode (w).

Of course, you may use multiple DRBD resources with a single DomU. In that case, simply add more entries like the one provided in the example to the disk option, separated by commas.

Under these circumstances, you must use the traditional phy: device syntax and the DRBD device name that is associated with your resource, not the resource name. That, however, requires that you manage DRBD state transitions outside Xen, which is a less flexible approach than that provided by the drbd resource type.

Once you have configured your DRBD-backed DomU, you may start it as you would any other DomU:

In the process, the DRBD resource you configured as the VBD will be promoted to the primary role, and made accessible to Xen as expected.

This is equally straightforward:

Again, as you would expect, the DRBD resource is returned to the secondary role after the DomU is successfully shut down.

This, too, is done using the usual Xen tools:

In this case, several administrative steps are automatically taken in rapid succession: * The resource is promoted to the primary role on destination-host. * Live migration of DomU is initiated on the local host. * When migration to the destination host has completed, the resource is demoted to the secondary role locally.

The fact that both resources must briefly run in the primary role on both hosts is the reason for having to configure the resource in dual-primary mode in the first place.

Xen supports two Virtual Block Device types natively:

This device type is used to hand “physical” block devices, available in the host environment, off to a guest DomU in an essentially transparent fashion.

This device type is used to make file-based block device images available to the guest DomU. It works by creating a loop block device from the original image file, and then handing that block device off to the DomU in much the same fashion as the phy device type does.

If a Virtual Block Device configured in the disk option of a DomU configuration uses any prefix other than phy:, file:, or no prefix at all (in which case Xen defaults to using the phy device type), Xen expects to find a helper script named block-prefix in the Xen scripts directory, commonly /etc/xen/scripts.

The DRBD distribution provides such a script for the drbd device type, named /etc/xen/scripts/block-drbd. This script handles the necessary DRBD resource state transitions as described earlier in this chapter.

To fully capitalize on the benefits provided by having a DRBD-backed Xen VBD’s, it is recommended to have Pacemaker manage the associated DomUs as Pacemaker resources.

You may configure a Xen DomU as a Pacemaker resource, and automate resource failover. To do so, use the Xen OCF resource agent. If you are using the drbd Xen device type described in this chapter, you will not need to configure any separate drbd resource for use by the Xen cluster resource. Instead, the block-drbd helper script will do all the necessary resource transitions for you.

When measuring the impact of using DRBD on a system’s I/O throughput, the absolute throughput the system is capable of is of little relevance. What is much more interesting is the relative impact DRBD has on I/O performance. Therefore, it is always necessary to measure I/O throughput both with and without DRBD.

I/O throughput estimation works by writing significantly large chunks of data to a block device, and measuring the amount of time the system took to complete the write operation. This can be easily done using a fairly ubiquitous utility, dd, whose reasonably recent versions include a built-in throughput estimation.

A simple dd-based throughput benchmark, assuming you have a scratch resource named test, which is currently connected and in the secondary role on both nodes, is one like the following:

This test simply writes 512MiB of data to your DRBD device, and then to its backing device for comparison. Both tests are repeated 5 times each to allow for some statistical averaging. The relevant result is the throughput measurements generated by dd.

See our Optimizing DRBD Throughput chapter for some performance numbers.

Latency measurements have objectives completely different from throughput benchmarks: in I/O latency tests, one writes a very small chunk of data (ideally the smallest chunk of data that the system can deal with), and observes the time it takes to complete that write. The process is usually repeated several times to account for normal statistical fluctuations.

Just as throughput measurements, I/O latency measurements may be performed using the ubiquitous dd utility, albeit with different settings and an entirely different focus of observation.

Provided below is a simple dd-based latency micro-benchmark, assuming you have a scratch resource named test which is currently connected and in the secondary role on both nodes:

This test writes 1,000 chunks with 4kiB each to your DRBD device, and then to its backing device for comparison. 4096 bytes is the smallest block size that a Linux system (on all architectures except s390), modern hard disks, and SSDs, are expected to handle.

It is important to understand that throughput measurements generated by dd are completely irrelevant for this test; what is important is the time elapsed during the completion of said 1,000 writes. Dividing this time by 1,000 gives the average latency of a single block write.

Furthermore, see our Optimizing DRBD Latency chapter for some typical performance values.

DRBD throughput is affected by both the bandwidth of the underlying I/O subsystem (disks, controllers, and corresponding caches), and the bandwidth of the replication network.

I/O subsystem throughput is determined, largely, by the number and type of storage units (disks, SSDs, other Flash storage [like FusionIO], …​) that can be written to in parallel. A single, reasonably recent, SCSI or SAS disk will typically allow streaming writes of roughly 40MiB/s to the single disk; an SSD will do 300MiB/s; one of the recent Flash storages (NVMe) will be at 1GiB/s. When deployed in a striping configuration, the I/O subsystem will parallelize writes across disks, effectively multiplying a single disk’s throughput by the number of stripes in the configuration. Therefore, the same 40MiB/s disks will allow effective throughput of 120MiB/s in a RAID-0 or RAID-1+0 configuration with three stripes, or 200MiB/s with five stripes; with SSDs, NVMe, or both, you can easily get to 1GiB/sec.

A RAID-controller with RAM and a BBU can speed up short spikes (by buffering them), and so too-short benchmark tests might show speeds like 1GiB/s too; for sustained writes its buffers will just run full, and then not be of much help, though.