Encrypted Replication With DRBD

With the arrival of DRBD® 9.2.6, replication traffic can now be encrypted by the Linux kernel. DRBD is now able to use in-kernel TLS acceleration (kTLS), which has been fully implemented since the release of Linux kernel version 4.17. LINBIT® developer Moritz Wanzenböck initially demoed the new feature during our recent community meeting.

While it has always been easy to configure disk encryption using dm-crypt with LINSTOR® and DRBD, the replication network traffic sent to the peer nodes was not easily encrypted. There were some existing solutions available for achieving encrypted network traffic such as implementing a VPN for use between nodes or sites, however, encryption was not natively configurable until now.

With DRBD 9.2.6, you can encrypt replication traffic with just a few configuration options.

Prerequisites

The following requirements are needed to enable encryption for each node:

• A somewhat modern Linux distribution such as Ubuntu 20.04 or RHEL 8 and newer providing Linux kernel version 4.17 or greater.
• DRBD version 9.2.6 or greater.
• drbd-utils version 9.26.0 or greater.
• Valid TLS certificates for encryption.
• The ktls-utils package installed.
• A properly configured DRBD resource file.

For help with installing and configuring DRBD, see our DRBD Users Guide.

TLS Certificates Deployment

For the test cluster used in this blog post, a private self-signed certificate authority (CA) was created and used to issue self-signed certificates for all nodes in the cluster.

Note that there are different methods and utilities used for generating TLS certificates, various security practices and organizational policies to consider, using public certificate authorities such as the extremely popular Let’s Encrypt, among other items to consider when deploying to production.

Generating TLS Certificates

First, the private certificate authority and all certificates will be generated from a client machine. One of the test nodes can also be used for this purpose. If you’re using the following Bash script snippet to generate your certificates, change the initial variables accordingly:

#!/bin/bash

# The CA name
ca_name="LINBIT Test CA"

# The CA filename
ca_filename="linbit_test_ca"

# Host certificates to generate
test_hosts=("alice" "bob" "charlie")

mkdir certificates && cd ./certificates

# Create private key for use with private CA using OpenSSL
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 \
-out $ca_filename.key

# Create a private CA certificate using OpenSSL
openssl req -key $ca_filename.key -new -x509 -days 3650 -subj "/CN=$ca_name" \
-out $ca_filename.crt

# Create certificates for each node
for host in ${test_hosts[@]}; do

    # Create a private key
    openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 \
    -out $host.key

    # Generate a CSR
    openssl req -key $host.key -subj "/CN=$host" -new -out $host.csr

    # Create a valid self-signed certificate the private CA
    openssl x509 -req -in $host.csr -CA $ca_filename.crt \
    -CAkey $ca_filename.key -CAcreateserial -days 365 -out $host.crt

    # Remove sequential serial number
    rm -f $ca_filename.srl
done

📝 NOTE: The above script and commands do not need to be run as root. The script will generate the corresponding files in a certificates directory:

```
$ ls -l | awk '{print $9}'

linbit_test_ca.crt
linbit_test_ca.key
alice.crt
alice.csr
alice.key
bob.crt
bob.csr
bob.key
charlie.crt
charlie.csr
charlie.key
```

Deploying TLS Certificates

1. Copy the CA certificate (linbit_test_ca.crt) to each node’s /etc/drbd.d/ directory.
2. Copy each node’s certificate (.crt file) and private key (.key file) to /etc/drbd.d/. The location for certificates is technically arbitrary depending on the file paths defined in /etc/tlshd.conf. For the sake of this blog post, they’re confined to DRBD’s default configuration directory (/etc/drbd.d/).
3. Change file permissions and ownership for the private keys and certificate on each node. This ensures that only the root user can view the contents of the private keys: # chmod 600 /etc/drbd.d/*.key` # chown root:root /etc/drbd.d/*.key # chown root:root /etc/drbd.d/*.crt
4. Store the certificate authority’s private key in a secure location.

Installing ktls-utils

DRBD relies on the ktls-utils package for performing TLS handshakes and session management inside user space before encryption can be performed in kernel space. There are two ways to obtain ktls-utils:

• For LINBIT customers: We now package ktls-utils in our LINBIT customer repositories. Simply install the package through your distribution’s package manager using dnf install ktls-utils or apt install ktls-utils.
• For non-LINBIT customers: Refer to the ktls-utils GitHub for building from source if it is not packaged by your Linux distribution.

Configuring ktls-utils

Once installed, ktls-utils contains a system-wide configuration file located in /etc/tlshd.conf. Each node needs to be configured with paths to the certificates and private keys in /etc/tlshd.conf. Here’s an example tlshd.conf from the node “alice”:

[authenticate.client]
x509.truststore=/etc/drbd.d/linbit_test_ca.crt
x509.certificate=/etc/drbd.d/alice.crt
x509.private_key=/etc/drbd.d/alice.key
[authenticate.server]
x509.truststore=/etc/drbd.d/linbit_test_ca.crt 
x509.certificate=/etc/drbd.d/alice.crt
x509.private_key=/etc/drbd.d/alice.k

After installation and configuration, the tlshd service needs to be started and enabled on each node:

# systemctl daemon-reload
# systemctl enable --now tlshd

Configuring DRBD for Encrypted Replication

Adding encryption to a DRBD configuration is as easy as adding "net { tls yes; }" to your DRBD configuration file. Included below is an example of a three node DRBD resource configuration files with encryption enabled:

resource r0 {
    device    /dev/drbd0;
    disk      /dev/vdb;
    meta-disk internal;

    net { tls yes; }

    on alice {
        address   192.168.222.20:7777;
        node-id   0;
    }

    on bob {
        address   192.168.222.21:7777;
        node-id   1;
    }

    on charlie {
        address   192.168.222.22:7777;
        node-id   2;
    }

    connection-mesh {
        hosts alice bob charlie;
    }
}

If encryption is being configured for newly defined DRBD resources, simply continue configuring DRBD by creating the metadata (`drbdadm create-md <resource>) and bringing up the DRBD virtual block device (drbdadm up <resource>).

If existing non-encrypted DRBD configurations are being updated to use encryption:

1. Update DRBD resource configuration files (.res files) with "net { tls yes; }" similar to the example above on all nodes.
2. Disconnect each DRBD resource by running drbdadm disconnect <resource> on all nodes.
3. Adjust each DRBD resource by running drbdadm adjust <resource> on all nodes.

Testing and Verifying Encryption

If encryption is being negotiated successfully, both tlshd and DRBD will log successful handshakes when connecting:

# journalctl --output cat | grep -i handshake
[...]
Handshake with bob (192.168.222.21) was successful
Handshake with charlie (192.168.222.22) was successful
drbd r0 bob: Handshake to peer 1 successful: Agreed network protocol version 122
drbd r0 charlie: Handshake to peer 2 successful: Agreed network protocol version 122
[...]

To verify encryption, simply capture network packets on a Secondary node using tcpdump while plain text data is being written on the Primary node:

# tcpdump port 7777 -i eth1 -A

dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
04:48:12.361639 IP 192.168.222.20.cbt > bob.49869: Flags [P.], seq 3189580912:3189583808, ack 609327991, win 501, options [nop,nop,TS val 986716313 ecr 3551259775], length 2896
E....+@.@.'..........a....$p$Q.w....H......:...........9z..../HCeH........s.."?9..T......v......._............;.......a@r..>&......+}.....9.S.k.&.fL.$..#7...C.h........'.ZR @i.@P.{..3K%V.7.....3N0.Q...B.SA....K...oA.._. ..}.........S..l._.L@rL!.Q..m?k......$.h./.K....|..I.\...:..j.;eEp.wXA....h.[...]

Additional Considerations

Adding encryption will diminish available performance and require additional CPU cycles. While a modern CPU with the Advanced Encryption Standard (AES) instruction set allows for more efficiency, there is still a performance hit. For that reason, you should benchmark your workloads in a testing environment before enabling encryption in production environments.

In an abstract sense, using encryption can also be viewed as sacrificing raw available performance (either throughput or IOPS) for security. Benchmarking encrypted workloads will be a future blog topic. In the meantime, feel free to share any benchmark results you might have from your own testing.

Conclusion

It has never been easier to encrypt replication traffic and we’re excited to release this new feature with DRBD 9.2.6. Happy encrypting with DRBD and kTLS!

As this feature is brand new to DRBD, it will soon be available in LINSTOR version 1.25. Stay tuned for the release notes and future announcements.