Recently I came across Synology’s spin on high availability. After perusing through their documentation I noticed an all too familiar protocol: DRBD®. Recognizing this software and having a recent desire to tinker a bit, I picked up two DS224+ hardware storage appliances, one of their least expensive models. LINBIT® is no stranger to assisting companies implement high availability or software-defined storage into their products. In fact, “back in my day” I helped several companies implement DRBD into their hardware and software appliances for high availability. This post explores behind-the-scenes on how it’s done and what software (or magic) they are using to make it all work.
The DS224+ runs Synology’s DiskStation Manager (DMS) software which provides a host of services. In addition to it being a NAS for backups, my personal Synology device runs their Photos app which acts as a Google Photos or iCloud replacement. There are many more things you can do with a Synology device, however that goes outside my reasoning for writing this blog.
With my pair of DS224+ devices in hand, I dug through my graveyard of parts that I just can’t seem to let go of and found four 2TB disks to populate the devices. Following the high availability wizard, I configured the Synology test units as a cluster. This only took about 20 minutes from start to finish. Synology’s documentation does a better job than I could do of explaining the “how to”, so head on over there if you’re interested. Below are my findings with regards to what software components they use to achieve high availability.
Replication
In Linux there are many ways to replicate your data across several nodes. Which tool you use depends on what your needs are and what features or functions are important to your deployment. In Synology’s case, they are using DRBD 8.4.11-1 for block level replication between devices. This is the version that is currently shipping in the Linux mainline kernel. DRBD is a smart choice when you have numerous ways to configure storage, as you do on these devices. It is high performing and has been used in production deployments for over two decades.
During the setup wizard there is the option of several file systems to choose from. Because DRBD is seen as a block device and is agnostic to the application or data layers above it, it doesn’t matter to DRBD what the choice is.
This is what the global DRBD configuration and resource configuration looks like on the DiskStation:
global {
usage-count no;
}
common {
net {
protocol C;
after-sb-0pri discard-zero-changes;
after-sb-1pri consensus;
after-sb-2pri disconnect;
timeout 40;
connect-int 30;
ping-int 5;
ping-timeout 50;
ko-count 105;
max-buffers 16K;
max-epoch-size 8K;
sndbuf-size 2M;
rcvbuf-size 2M;
}
disk {
c-max-rate 4096M;
resync-rate 4096M;
c-plan-ahead 10;
c-min-rate 105M;
c-fill-target 500M;
disk-flushes no;
md-flushes no;
on-io-error pass_on;
fencing resource-only;
}
handlers {
split-brain "/var/packages/HighAvailability/target/etc/shaessential.sh split-brain-handler";
fence-peer "/var/packages/HighAvailability/target/etc/shaessential.sh fence-peer-handler";
}
}
# resource vg1-volume_1 on synology-ha: not ignored, not stacked
# defined at /usr/syno/etc/packages/HighAvailability/drbd.d/vg1-volume_1.res:1
resource vg1-volume_1 {
floating ipv4 169.254.1.1:7400 {
device /dev/drbd-vg1-volume_1 minor 0;
disk /dev/vg1/volume_1;
meta-disk /dev/loop.vg1-volume_1;
}
floating ipv4 169.254.1.2:7400 {
device /dev/drbd-vg1-volume_1 minor 0;
disk /dev/vg1/volume_1;
meta-disk /dev/loop.vg1-volume_1;
}
net {
cram-hmac-alg sha1;
shared-secret 3176460778;
}
}
I’ve never seen external metadata on a loop device, so I had to do some digging to figure out what this is about. After some help from some colleagues we sorted it out. Interesting stuff… the DRBD metadata is created on the root volume via a loop device and not the user created volumes, as seen below.
root@synology-ha:/usr/local/packages/@appstore/HighAvailability/etc# losetup --list
NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC
/dev/loop0 0 0 0 0 /var/lib/drbd_md/drbd_md.vg1-volume_1 0 512
By using external metadata in this way, it enables Synology to add high availability after the fact on existing volumes or devices. They also avoid having to resize the file system or worry about tuning the metadata to accommodate various RAID configurations. Pretty clever. More information on comparing internal and external metadata can be found in the DRBD User’s Guide.
Also interesting is the use of floating peers in the configuration file. By using floating peers, DRBD peers are not tied to specific named hosts, as they might be in a conventional configuration. Rather a DRBD resource configured this way can float between several hosts. In a floating peers configuration, DRBD identifies peers by virtual IP addresses, rather than by hostname. This allows the hostnames to be changed without having to update the resource configuration files. This is important for the Synology use case, where the user can change the hostname of the NAS appliance. The use of virtual IP addresses is also important here for a similar reason.
Another configuration aspect to mention is that Synology uses symbolic links to make the data from their applications highly available. This is a relatively common approach and avoids having to modify configuration files or default paths to achieve high availability. Here you can see an example of the Synology Photos application symlinks to the DRBD-replicated volume (/volume1).
root@synology-ha:/var/packages/SynologyPhotos# ls -alh
total 80K
drwxr-xr-x 5 root root 4.0K Feb 28 07:58 .
drwxr-xr-x 21 root root 4.0K Feb 27 17:07 ..
drwxr-xr-x 3 root root 4.0K Feb 28 07:58 conf
-rw-r--r-- 1 root root 0 Feb 28 07:58 enabled
lrwxrwxrwx 1 root root 32 Feb 26 16:34 etc -> /volume1/@appconf/SynologyPhotos
lrwxrwxrwx 1 root root 32 Feb 26 16:34 home -> /volume1/@apphome/SynologyPhotos
-rw-r--r-- 1 root root 59K Feb 26 16:34 INFO
drwxr-xr-x 5 root root 4.0K Feb 26 16:34 scripts
lrwxrwxrwx 1 root root 33 Feb 26 16:34 share -> /volume1/@appshare/SynologyPhotos
lrwxrwxrwx 1 root root 33 Feb 26 16:34 target -> /volume1/@appstore/SynologyPhotos
lrwxrwxrwx 1 root root 32 Feb 26 16:34 tmp -> /volume1/@apptemp/SynologyPhotos
lrwxrwxrwx 1 root root 32 Feb 26 16:34 var -> /volume1/@appdata/SynologyPhotos
drwxr-xr-x 2 root root 4.0K Feb 26 16:34 WIZARD_UIFILES
root@synology-ha:/var/packages/SynologyPhotos#
Services
Once you enable high availability, managing the Synology storage device is done through a web console on a virtual IP address that is activated on whatever host is the current primary server.
At first glance it doesn’t appear to be too complex of a high availability deployment, in fact it seems a little too simple:
root@synology-ha:~# crm_mon -n1
Stack: corosync
Current DC: syn1 (version 1.1.19-e141164) - partition with quorum
Last updated: Wed Feb 28 10:00:58 2024
Last change: Wed Feb 28 07:57:50 2024 by root via cibadmin on syn2
2 nodes configured
12 resources configured
Node syn1: online
PING_SERVER_ (ocf::synology:ping): Started
DRBD_ (ocf::synology:drbd): Slave
DSM_REAR_ (ocf::synology:dsm_rear): Slave
DSM_FIRST_ (ocf::synology:dsm_first): Slave
Node syn2: online
PROMOTE_BEGIN (ocf::synology:PromoteBegin): Started
DUMMY_START (ocf::synology:Dummy): Started
PING_SERVER_ (ocf::synology:ping): Started
DRBD_ (ocf::synology:drbd): Master
DSM_FIRST_ (ocf::synology:dsm_first): Master
DSM_REAR_ (ocf::synology:dsm_rear): Master
NETWORK (ocf::synology:Network): Started
DUMMY_END (ocf::synology:Dummy): Started
These resources consist of custom OCF agents located in:
root@synology-ha:/usr/local/packages/@appstore/HighAvailability/lib/ocf/resource.d/synology# ls
drbd dsm_first dsm_rear Dummy Network ping PromoteBegin
After some digging I discovered a directory full of shell scripts. This is where the real magic happens. There are numerous scripts that manage the preparation, startup, shutdown, monitoring, upgrading, and other operations of the Synology cluster. These scripts are called at various times to ensure management and functionality is as seamless as it should be in an appliance such as this.
The virtual IP address, along with the DRBD devices, are managed through a combination of Pacemaker and the aforementioned scripts. Pacemaker promotes the various resources, which then tell the system where to run the scripts. Those scripts are executed by the system through systemd unit files and are located throughout the /usr/local/packages/@appstore/HighAvailability/ directory.
Whoops! An Accidental Discovery
During a meeting with the sales team, I lifted up one of the Synology devices to show the team and accidentally turned it on. I let the machine boot because I didn’t want to hard power it off mid-boot. To my surprise the other device started booting. After doing some testing I discovered that within one of these scripts is a call that will send Wake-on-LAN packets out both network interfaces to wake the other node. I also found some configuration and scripting magic that makes it so that services also prefer to run on the node with the most connectivity (both interfaces are connected rather than just one). Pretty cool.
Parsing through the various scripts, reveals another directory. The rabbit hole continues over at:
/usr/syno/etc/packages/HighAvailability/
This directory houses the individual DRBD resource and cluster configuration files. In particular, the ha.cf file is interesting – historically this is where the Heartbeat configuration is defined. Synology, however, uses the ha.cf to configure cluster variables used by the scripts mentioned above. These are variables such as the floating IP addresses for DRBD replication, public facing IP addresses of the Synology devices, and the management console virtual IP address. None of the parameters found in the ha.cf man page are located in this file which leads me to believe that this is a leftover relic from their move from using Heartbeat to using Pacemaker and Corosync. That’s pure speculation and might be wrong. Perhaps it’s a coincidence and they just wanted to name the file ha.cf.
To manage the available nodes, Synology uses Corosync. The configuration file includes two network rings: one on the public interface, and one on the crossover (replication) interface. This helps ensure that failovers only happen when a node is truly down.
Conclusion
Open source is something I’ve been passionate about since the late 1900s. Stumbling across things like this reinforces that interest. It allows companies and individuals to have the same power to create and deploy enterprise grade solutions without the enterprise cost. When companies use open source software to do unique things, such as Synology using DRBD in the NAS appliance investigated here, it has the power to advance the software project or influence its direction in new ways. Better yet, because it’s open source, companies can choose to use the software with or without the developers, with or without support. Options are always nice to have.
If you’ve opted for the Synology high availability route, chances are you’re not overly interested in manually configuring or managing a high availability system through the command line. Configuring high availability can be a complex task. Synology has managed to make it extremely easy. While their take is unique and not one I’ve seen before, it seems to work and work well at that. Between the numerous scripts they use and the Wake-on-LAN functionality to wake the peer up when powering on the units, I was quite impressed. I also learned about using a loop device for external metadata to add DRBD after the fact without having to resize the file system. For anyone wanting a “just works” solution, this might be your ticket.
If you’re interested in integrating DRBD or LINSTOR® into your hardware or software appliance please feel free to reach out. Our engineers and solutions architects are available to bounce ideas of off. We always like to hear what the world is doing with our software. No commitments required, in the true open source spirit.
