NAS High Availability Using a Synology NAS Appliance & LINBIT Software

Recently I came across Synology’s spin on high availability. After perusing through their documentation I noticed an all too familiar protocol: DRBD®. Recognizing this software and having a recent desire to tinker a bit, I picked up two DS224+ hardware storage appliances, one of their least expensive models. LINBIT® is no stranger to assisting companies implement high availability or software-defined storage into their products. In fact, “back in my day” I helped several companies implement DRBD into their hardware and software appliances for high availability. This post explores behind-the-scenes on how NAS high availability is done and what software (or magic) Synology is using to make it all work.

The DS224+ runs Synology’s DiskStation Manager (DMS) software which provides a host of services. In addition to it being a NAS for backups, my personal Synology device runs their Photos app which acts as a Google Photos or iCloud replacement. There are many more things you can do with a Synology device, however that goes outside my reasoning for writing this blog.

With my pair of DS224+ devices in hand, I dug through my graveyard of parts that I just can’t seem to let go of and found four 2TB disks to populate the devices. Following the high availability wizard, I configured the Synology test units as a cluster. This only took about 20 minutes from start to finish. Synology’s documentation does a better job than I could do of explaining the “how to”, so head on over there if you’re interested. Below are my findings with regards to what software components they use to achieve high availability.

Replication

In Linux there are many ways to replicate your data across several nodes. Which tool you use depends on what your needs are and what features or functions are important to your deployment. In Synology’s case, they are using DRBD 8.4.11-1 for block level replication between devices. This is the version that is currently shipping in the Linux mainline kernel. DRBD is a smart choice when you have numerous ways to configure storage, as you do on these devices. It is high performing and has been used in production deployments for over two decades.

During the setup wizard there is the option of several file systems to choose from. Because DRBD is seen as a block device and is agnostic to the application or data layers above it, it doesn’t matter to DRBD what the choice is.

This is what the global DRBD configuration and resource configuration looks like on the DiskStation:

global {
    usage-count no;
}

common {
    net {
        protocol           C;
        after-sb-0pri    discard-zero-changes;
        after-sb-1pri    consensus;
        after-sb-2pri    disconnect;
        timeout           40;
        connect-int       30;
        ping-int           5;
        ping-timeout      50;
        ko-count         105;
        max-buffers      16K;
        max-epoch-size    8K;
        sndbuf-size       2M;
        rcvbuf-size       2M;
    }
    disk {
        c-max-rate       4096M;
        resync-rate      4096M;
        c-plan-ahead      10;
        c-min-rate       105M;
        c-fill-target    500M;
        disk-flushes      no;
        md-flushes        no;
        on-io-error      pass_on;
        fencing          resource-only;
    }
    handlers {
        split-brain      "/var/packages/HighAvailability/target/etc/shaessential.sh split-brain-handler";
        fence-peer       "/var/packages/HighAvailability/target/etc/shaessential.sh fence-peer-handler";
    }
}

# resource vg1-volume_1 on synology-ha: not ignored, not stacked
# defined at /usr/syno/etc/packages/HighAvailability/drbd.d/vg1-volume_1.res:1
resource vg1-volume_1 {
    floating ipv4 169.254.1.1:7400 {
        device           /dev/drbd-vg1-volume_1 minor 0;
        disk             /dev/vg1/volume_1;
        meta-disk        /dev/loop.vg1-volume_1;
    }
    floating ipv4 169.254.1.2:7400 {
        device           /dev/drbd-vg1-volume_1 minor 0;
        disk             /dev/vg1/volume_1;
        meta-disk        /dev/loop.vg1-volume_1;
    }
    net {
        cram-hmac-alg    sha1;
        shared-secret    3176460778;
    }
}

I’ve never seen external metadata on a loop device, so I had to do some digging to figure out what this is about. After some help from some colleagues we sorted it out. Interesting stuff… the DRBD metadata is created on the root volume via a loop device and not the user created volumes, as seen below.

root@synology-ha:/usr/local/packages/@appstore/HighAvailability/etc# losetup --list
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                             DIO LOG-SEC
/dev/loop0         0      0         0  0 /var/lib/drbd_md/drbd_md.vg1-volume_1   0     512

By using external metadata in this way, it enables Synology to add high availability after the fact on existing volumes or devices. They also avoid having to resize the file system or worry about tuning the metadata to accommodate various RAID configurations. Pretty clever. More information on comparing internal and external metadata can be found in the DRBD User’s Guide.

Also interesting is the use of floating peers in the configuration file. By using floating peers, DRBD peers are not tied to specific named hosts, as they might be in a conventional configuration. Rather a DRBD resource configured this way can float between several hosts. In a floating peers configuration, DRBD identifies peers by virtual IP addresses, rather than by hostname. This allows the hostnames to be changed without having to update the resource configuration files. This is important for the Synology use case, where the user can change the hostname of the NAS appliance. The use of virtual IP addresses is also important here for a similar reason.

Another configuration aspect to mention is that Synology uses symbolic links to make the data from their applications highly available. This is a relatively common approach and avoids having to modify configuration files or default paths to achieve high availability. Here you can see an example of the Synology Photos application symlinks to the DRBD-replicated volume (/volume1).

root@synology-ha:/var/packages/SynologyPhotos# ls -alh
total 80K
drwxr-xr-x  5 root root 4.0K Feb 28 07:58 .
drwxr-xr-x 21 root root 4.0K Feb 27 17:07 ..
drwxr-xr-x  3 root root 4.0K Feb 28 07:58 conf
-rw-r--r--  1 root root    0 Feb 28 07:58 enabled
lrwxrwxrwx  1 root root   32 Feb 26 16:34 etc -> /volume1/@appconf/SynologyPhotos
lrwxrwxrwx  1 root root   32 Feb 26 16:34 home -> /volume1/@apphome/SynologyPhotos
-rw-r--r--  1 root root  59K Feb 26 16:34 INFO
drwxr-xr-x  5 root root 4.0K Feb 26 16:34 scripts
lrwxrwxrwx  1 root root   33 Feb 26 16:34 share -> /volume1/@appshare/SynologyPhotos
lrwxrwxrwx  1 root root   33 Feb 26 16:34 target -> /volume1/@appstore/SynologyPhotos
lrwxrwxrwx  1 root root   32 Feb 26 16:34 tmp -> /volume1/@apptemp/SynologyPhotos
lrwxrwxrwx  1 root root   32 Feb 26 16:34 var -> /volume1/@appdata/SynologyPhotos
drwxr-xr-x  2 root root 4.0K Feb 26 16:34 WIZARD_UIFILES
root@synology-ha:/var/packages/SynologyPhotos#

Services

Once you enable high availability, managing the Synology storage device is done through a web console on a virtual IP address that is activated on whatever host is the current primary server.

At first glance it doesn’t appear to be too complex of a high availability deployment, in fact it seems a little too simple:

root@synology-ha:~# crm_mon -n1
Stack: corosync
Current DC: syn1 (version 1.1.19-e141164) - partition with quorum
Last updated: Wed Feb 28 10:00:58 2024
Last change: Wed Feb 28 07:57:50 2024 by root via cibadmin on syn2

2 nodes configured
12 resources configured

Node syn1: online
       PING_SERVER_    (ocf::synology:ping):   Started
       DRBD_   (ocf::synology:drbd):   Slave
       DSM_REAR_       (ocf::synology:dsm_rear):       Slave
       DSM_FIRST_      (ocf::synology:dsm_first):      Slave
Node syn2: online
       PROMOTE_BEGIN   (ocf::synology:PromoteBegin):   Started
       DUMMY_START     (ocf::synology:Dummy):  Started
       PING_SERVER_    (ocf::synology:ping):   Started
       DRBD_   (ocf::synology:drbd):   Master
       DSM_FIRST_      (ocf::synology:dsm_first):      Master
       DSM_REAR_       (ocf::synology:dsm_rear):       Master
       NETWORK (ocf::synology:Network):        Started
       DUMMY_END       (ocf::synology:Dummy):  Started

These resources consist of custom OCF agents located in:

root@synology-ha:/usr/local/packages/@appstore/HighAvailability/lib/ocf/resource.d/synology# ls
drbd  dsm_first  dsm_rear  Dummy  Network  ping  PromoteBegin

After some digging I discovered a directory full of shell scripts. This is where the real magic happens. There are numerous scripts that manage the preparation, startup, shutdown, monitoring, upgrading, and other operations of the Synology cluster. These scripts are called at various times to ensure management and functionality is as seamless as it should be in an appliance such as this.

The virtual IP address, along with the DRBD devices, are managed through a combination of Pacemaker and the aforementioned scripts. Pacemaker promotes the various resources, which then tell the system where to run the scripts. Those scripts are executed by the system through systemd unit files and are located throughout the /usr/local/packages/@appstore/HighAvailability/ directory.

Whoops! An Accidental Discovery

During a meeting with the sales team, I lifted up one of the Synology devices to show the team and accidentally turned it on. I let the machine boot because I didn’t want to hard power it off mid-boot. To my surprise the other device started booting. After doing some testing I discovered that within one of these scripts is a call that will send Wake-on-LAN packets out both network interfaces to wake the other node. I also found some configuration and scripting magic that makes it so that services also prefer to run on the node with the most connectivity (both interfaces are connected rather than just one). Pretty cool.

Parsing through the various scripts, reveals another directory. The rabbit hole continues over at:

/usr/syno/etc/packages/HighAvailability/

This directory houses the individual DRBD resource and cluster configuration files. In particular, the ha.cf file is interesting – historically this is where the Heartbeat configuration is defined. Synology, however, uses the ha.cf to configure cluster variables used by the scripts mentioned above. These are variables such as the floating IP addresses for DRBD replication, public facing IP addresses of the Synology devices, and the management console virtual IP address. None of the parameters found in the ha.cf man page are located in this file which leads me to believe that this is a leftover relic from their move from using Heartbeat to using Pacemaker and Corosync. That’s pure speculation and might be wrong. Perhaps it’s a coincidence and they just wanted to name the file ha.cf.

To manage the available nodes, Synology uses Corosync. The configuration file includes two network rings: one on the public interface, and one on the crossover (replication) interface. This helps ensure that failovers only happen when a node is truly down.

Conclusion

Open source is something I’ve been passionate about since the late 1900s. Stumbling across things like this reinforces that interest. It allows companies and individuals to have the same power to create and deploy enterprise grade solutions without the enterprise cost. When companies use open source software to do unique things, such as Synology using DRBD in the NAS appliance investigated here, it has the power to advance the software project or influence its direction in new ways. Better yet, because it’s open source, companies can choose to use the software with or without the developers, with or without support. Options are always nice to have.

If you’ve opted for the Synology high availability route, chances are you’re not overly interested in manually configuring or managing a high availability system through the command line. Configuring high availability can be a complex task. Synology has managed to make it extremely easy. While their take is unique and not one I’ve seen before, it seems to work and work well at that. Between the numerous scripts they use and the Wake-on-LAN functionality to wake the peer up when powering on the units, I was quite impressed. I also learned about using a loop device for external metadata to add DRBD after the fact without having to resize the file system. For anyone wanting a “just works” solution, this might be your ticket.

If you’re interested in integrating DRBD or LINSTOR® into your hardware or software appliance please feel free to reach out. Our engineers and solutions architects are available to bounce ideas of off. We always like to hear what the world is doing with our software. No commitments required, in the true open source spirit.

Brian Hellman

Brian Hellman

Brian started LINBIT USA in 2008 and leads its dedicated team as Chief Operating Officer. Brian is passionate about open source software and the community around it. He is particularly focused on high availability, disaster recovery, and software-defined storage technologies that help empower organizations to create customizable solutions and allow them to preserve control of their data. Outside of technological interests, Brian is a dedicated philanthropist through Freemasonry, The Scottish Rite, and The Shriner's Hospital for Children.

Talk to us

LINBIT is committed to protecting and respecting your privacy, and we’ll only use your personal information to administer your account and to provide the products and services you requested from us. From time to time, we would like to contact you about our products and services, as well as other content that may be of interest to you. If you consent to us contacting you for this purpose, please tick above to say how you would like us to contact you.

You can unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.

By clicking submit below, you consent to allow LINBIT to store and process the personal information submitted above to provide you the content requested.

Talk to us

LINBIT is committed to protecting and respecting your privacy, and we’ll only use your personal information to administer your account and to provide the products and services you requested from us. From time to time, we would like to contact you about our products and services, as well as other content that may be of interest to you. If you consent to us contacting you for this purpose, please tick above to say how you would like us to contact you.

You can unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.

By clicking submit below, you consent to allow LINBIT to store and process the personal information submitted above to provide you the content requested.