Over its more than 20 years of existence, DRBD® has earned some renown as data replication software with high performance. Put to the test on some potent hardware, the LINBIT® team has shown impressive benchmarking numbers1. Independent testing has also shown strong results on more conventional hardware2. Performance is one reason DRBD is often used to help make applications, especially applications such as databases and messaging queues that require small and frequent read/write operations, highly available.
DRBD memory consumption
While the highlights of DRBD performance testing have been high IOPS results and low CPU consumption, memory usage is typically a secondary consideration. This is likely because benchmarking often happens in a single 3-node cluster, and might typically involve just a single storage resource.
DRBD memory usage is proportional to the size of replicated storage. Knowing the size of your replicated storage, it is simple to estimate how much memory a storage node will need for DRBD3.
With version 9, DRBD needs about 32 MiB of memory for each TiB of replicated storage per peer with backing storage. A storage offering of 64TiB, made highly available by DRBD, for example, would need 2GiB of memory on each storage node.
This is all to say that there are no surprises or knobs to turn to change DRBD memory consumption. That is, there were no knobs to turn before the development work on a DRBD feature that is the focus of this article.
The impact of DRBD memory requirements on large enterprises
32MiB per TiB of replicated storage might not seem like much, but as you scale your replicated storage, DRBD memory usage will become a consideration at some point. Large enterprises such as hosting or cloud service providers might feel this impact first. They are often not only scaling up but scaling out. Imagine a service provider extending that earlier mentioned 64TiB storage offering to hundreds of customers, or thousands, or more, and how much memory the enterprise will need for its highly available storage.
Particularly attention-grabbing to these enterprises will be the memory costs of scaling out. This is where LINBIT developers have stepped in to improve the way DRBD uses memory by giving users flexibility to influence how much information DRBD stores in memory. This is also at the request, and thanks to the support, of a large enterprise customer which has already considered the memory impact to its operations and scaling.
Technical details about improving DRBD memory consumption
Before discussing the nature of DRBD development improvements around memory usage, some background on what DRBD uses memory for will give some context. The main source of DRBD memory usage is a change-tracking bitmap. DRBD sets a single bit of memory whenever a block of replicated storage changes on only one of the local disk or the peer’s disk. The most common scenario is when a block is written while a peer node is disconnected, for example, because of a network issue or a backing disk failure. After the disconnected node reconnects, DRBD can use the change-tracking bitmap stored in memory to decide which storage blocks have had changes since the peer disconnected. DRBD will then synchronize only those blocks, rather than synchronizing the entire storage. This is why the DRBD bitmap is sometimes called the quick-sync bitmap.
At the time of writing, the latest version of DRBD (9.2) uses a block size of 4KiB for the DRBD change-tacking bitmap. As a side mention, DRBD also has a version of the bitmap stored on-disk in the activity log within DRBD metadata. Another technical point is that DRBD allocates a change-tracking bitmap even if there is no defined peer for data replication. This has implications for service providers who might have single availability zone deployments. I will return to this consideration later.
The solution
A new feature in DRBD 9.3 gives users the ability to specify the block size that the bitmap uses. This flexibility takes the form of a --bitmap-granularity option to the drbdadm create-metadata command. With this option, users can select a bitmap granularity between 4KiB and 1MiB, in successive powers of two.
DRBD peers can connect normally even when their bitmap granularity sizes are different. This makes it possible to migrate a cluster to a different bitmap granularity size. In normal usage, it is recommended that nodes in a cluster use a common bitmap granularity size.
The effects of bitmap granularity on memory consumption
As mentioned earlier, DRBD 9.2 has an unchangeable bitmap granularity of 4KiB. Beginning with DRBD 9.3, increasing the bitmap granularity value will reduce DRBD memory consumption in an inversely proportional manner.
The following table shows the effects of different bitmap granularity values on DRBD memory consumption. Thanks to the power of inverse exponential proportionality, the table shows drastic reductions in DRBD memory usage with increases to the DRBD bitmap granularity.
| DRBD bitmap granularity (KiB) | DRBD memory usage (KiB) per 1 TiB storage |
|---|---|
| 4 | 32,768 |
| 8 | 16,384 |
| 16 | 8,192 |
| 32 | 4,096 |
| 64 | 2,048 |
| 128 | 1,024 |
| 256 | 512 |
| 512 | 256 |
| 1024 | 128 |
Another DRBD feature benefit for service providers
The main benefit to being able to specify different DRBD bitmap granularity values is so that users can reduce DRBD memory requirements and save the associated hardware costs. As mentioned earlier, DRBD allocates memory for a storage resource’s change-tracking bitmap, even when there is no peer defined for the resource. This might be the case for a service provider where a storage resource is only deployed in a single availability zone. A DRBD peer in this case might only come into play for a service provider when a customer requests storage in multiple availability zones.
With the bitmap granularity feature in DRBD 9.3, LINBIT developers also introduced a DRBD resource configuration option so users can choose not to use a change-tracking bitmap. This option is only allowed when the resource has no defined peers. DRBD replication in this scenario is “waiting in the wings” but users have the flexibility to not have a DRBD bitmap consuming memory while in this peerless state.
The tradeoff
While selecting larger bitmap granularity values reduces DRBD memory requirements, there are some caveats. A larger bitmap granularity value means the bitmap uses a larger block size. This means DRBD tracks larger areas on replicated storage for changes. If DRBD primary storage data changes while a DRBD peer is in a disconnected state, the entire area of coverage containing those changes will synchronize when the peer reconnects. The implication is that a change of even a few KiBs of data might cause MiBs of synchronization to happen when a peer node reconnects. This can mean synchronizing takes longer after recovering from failure events which could affect a recovery time objective for a disconnected node.
Conclusion
While DRBD memory requirements are modest, they can compound as an organization scales its replicated storage up and out. Cloud hosting and other service providers offering storage, or services dependent on storage, are particularly exposed to this. The motivating factor behind the DRBD bitmap granularity feature is so users can reduce DRBD memory requirements on their storage nodes. By selecting larger bitmap granularity values, an organization can reduce the amount of memory it needs to run DRBD, and save money in hardware costs as it scales its storage offerings.
- Benchmarking on Ampere Altra Max Platform with LINBIT SDS↩︎
- Independent testers, such as StarWind and Andrei Kvapil of CozyStack have also had good results performance testing DRBD and LINSTOR®, even on more modest setups than the Ampere platform.↩︎
- To calculate exact or approximate memory requirements for DRBD, refer to the formulas in the DRBD 9 User Guide.↩︎
