Introducing DRBD Proxy version 4

In January 2025, LINBIT® quietly released DRBD Proxy version 4 (or, DRBD Proxy v4, for short). For those who are not familiar with DRBD Proxy, it is a transparent proxy for DRBD® connections that buffers and compresses replicated data between DRBD hosts. It is often used by LINBIT customers who need to replicate their DRBD devices across very long distances. This typically implies that high latency and low bandwidth networks are between the DRBD peers.

DRBD Proxy’s buffering allows a configurable amount of system memory to be used as buffer space for holding replicated data, which flushes as fast as the replication network can handle. This allows the DRBD device to continue accepting new writes without having to block I/O when the relatively small Linux kernel TCP send buffers become full.

DRBD Proxy’s tunable compression levels allow users to trade CPU cycles for more efficient use of network bandwidth when possible. With certain datasets, such as text-based data (txt, csv, JSON, xml, office documents), database dumps, and log files, compression can save huge amounts of bandwidth with relatively low CPU utilization.

/me puts sales hat on

DRBD Proxy is one of the few closed source projects that LINBIT develops, so you will need to obtain a license from LINBIT before you can use it. However, LINBIT offers 30-day evaluations that come with email support direct from LINBIT engineers and developers. If you’re intrigued and want to test DRBD Proxy version 4 in your own environment, or maybe you just have questions you’d like to ask the LINBIT team directly, reach out to us and we’ll happily oblige.

/me takes off sales hat

But maybe you’re still here because you just want to read about what’s new before jumping into any serious relationship with LINBIT. That’s OK too.

What’s new in DRBD Proxy v4

Much has changed between DRBD Proxy v3 and v4, mostly internally, but that translates to user-facing differences we hope our users will appreciate. With that said, it should also be mentioned that DRBD Proxy v3 configurations are compatible with DRBD Proxy v4, so moving to DRBD Proxy v4 should be relatively simple.

DRBD Proxy v4 is written in Rust

DRBD Proxy v4 is the first new major version of DRBD Proxy in over a decade. It’s not just an update though, it’s a complete rewrite in a different programming language than its predecessor. This is the largest difference and could be the most important.

DRBD Proxy v3 was written in C, a very fast and low-level language that pairs well with DRBD and DRBD userland utilities also written in C. The C programming language, however, requires much attention to detail when it comes to memory and thread handling, leaving room for developers to introduce security issues and race conditions if they’re not careful. This can be seen as a hurdle for developers without extensive experience working with C.

DRBD Proxy v4 is now written in Rust, a modern programming language that is considered to be equally fast and low-level when compared to C. Rust has built-in memory management and built-in threading (std::thread) making it safer and more approachable when developing, without sacrificing much control.

This all adds up to making DRBD Proxy easier to maintain and improve upon.

Quicker to connect DRBD peers

While it might not be apparent from a user’s perspective, connecting DRBD peers, especially through an intermediary process such as DRBD Proxy, is relatively complex. There are hints of this complexity in how DRBD Proxy connections are configured in the DRBD resource configuration file. For example, examine this relatively simple 2-node DRBD Proxy enabled DRBD resource configuration:

resource r0 {
    protocol A;
    device minor 0;
    disk /dev/vg_drbd/lv_r0;
    meta-disk internal;

    proxy {
        memlimit 200M;
        plugin {
            zstd level 3;
        }
    }

    on alpha {
        address 127.0.0.1:7789;
        proxy on alpha {
            inside 127.0.0.1:7788;
            outside 192.168.6.14:7790;
        }
    }

    on bravo {
        address 127.0.0.1:7789;
        proxy on bravo {
            inside 127.0.0.1:7788;
            outside 192.168.7.15:7790;
        }
    }
}

Notice that there are three different network sockets defined for each of the DRBD hosts, alpha and bravo. Each of those sockets are involved in establishing a connection between DRBD device peers.

DRBD Proxy v4 lead developer, Joel Colledge, considered multiple different strategies for establishing connections, each with varying connection speeds when used in environments with different latency profiles. Ultimately, the connection sequence that was chosen, connected the quickest under almost all circumstances, which happens to be much quicker than DRBD Proxy v3 connection time in almost all cases.

Simplified compression configurations

DRBD Proxy v3 was developed to support many compression algorithms by using different compression plugins. At the time, this was important because different datasets and algorithms had different requirements and characteristics that worked better in certain combinations.

Since then, Zstandard (zstd) was developed by Yann Collet and provides compression ratios close to lzma at its highest compression levels, while also having one of the fastest decoders among current compression algorithms. Performance numbers and considerations are available on the Zstandard webpage.

Because Zstandard supports a wide range of compression levels, all with very good compression and decompression speeds for their compression ratio, this is the only compression algorithm supported in DRBD Proxy v4. DRBD Proxy v4 allows users to configure zstd compression levels ranging from 1 to 22, which enables fine grained control when finding the CPU to compression ratio “sweet spot”.

Compression is enabled in the plugin section of the DRBD Proxy configuration, and can be tuned by adjusting the level up or down, with 1 being the lowest compression and 22 being the highest:

resource r0 {
[...]
    proxy {
        memlimit 200M;
        plugin {
            zstd level 3;
        }
    }
[...]
}

💡 TIP: To make DRBD Proxy v4 compatible with DRBD Proxy v3 configurations, DRBD Proxy v4 will silently convert zlib and lz4 compression plugin configurations that were valid for DRBD Proxy v3 into the equivalent zstd compression level.

You can verify how well your data is being compressed by writing some test data, which should be representative of your actual data type, into the DRBD device and reviewing the CompressionRatio using the drbd-proxy-ctl command line utility as shown below:

## Check the compression level:
[root@charlie drbd_mount]# drbd-proxy-ctl list plugins
connection r0-u-alpha_bravo-charlie:
    zstd level 3

## Write test data to the DRBD device (example is War and Peace by Leo Tolstoy)
[root@charlie drbd_mount]# curl -LO https://www.gutenberg.org/ebooks/2600.txt.utf-8
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   302  100   302    0     0    850      0 --:--:-- --:--:-- --:--:--   848
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 3280k  100 3280k    0     0  2075k      0  0:00:01  0:00:01 --:--:-- 4846k

## Check the compression ratio
[root@charlie drbd_mount]# drbd-proxy-ctl show hconnections
 Name                      Status     Inside    Outside  CompressionRatio
 r0-u-alpha_bravo-charlie  Connected  59.24KiB  4.36KiB  14.690

The plain text copy of War and Peace was able to be compressed down to about 8% of its original size by DRBD Proxy, saving much bandwidth on the replication network.

Concluding thoughts

DRBD Proxy version 4 is available to LINBIT customers that are using Linux distributions that package kernel version 5.1 or newer. A full how-to guide on configuring and using DRBD Proxy v4 is available on LINBIT’s documentation page, which covers configuring and tuning DRBD and DRBD Proxy in more detail.

Outside of the more prominent changes mentioned in the sections above, getting information about DRBD Proxy v4 connections using drbd-proxy-ctl was also enhanced and is touched on more in the technical guide linked above. For example, watching the DRBD Proxy buffer fill and flush alongside DRBD’s verbose statistics can be useful when determining DRBD Proxy buffer sizes, and is more detailed than it was in DRBD Proxy v3. This is apparent from the output of the shell command used in the bottom portion of the GIF below, while true; do drbdadm status --statistics --verbose; drbd-proxy-ctl show memusage; sleep 0.25s; clear; done, during a dd write operation to a file system mount backed by the DRBD device, shown in the top portion of the GIF:

Play with DRBD Proxy v4 on your own systems and explore the utilities and improvements it brings to DRBD’s asynchronous replication. If you’re interested in testing or just talking about DRBD Proxy and how you might use it to replicate data between geographically distributed systems, don’t hesitate to reach out to LINBIT.