Highly Available LINSTOR Controller with Pacemaker

*UPDATE MAY 2022: This article was originally published back in 2018. Three years before our DRBD-Reactor project. With the release of the DRBD-Reactor software we now suggest using that to make a Highly available LINSTOR® Controller. Instructions for this can be found in our User’s Guide. However, using Pacemaker is still a valid option. So this post will remain listed at present.


Part of the design of LINSTOR is that if the central LINSTOR Controller goes down, all the storage still remains up and accessible. This should allow ample time to repair the downed system hosting the LINSTOR Controller. Still, in the majority of cases, it is preferred to run the LINSTOR Controller in a container within your cloud or as a VM in your hypervisor platform. However, there may exist a situation where you want to keep the LINSTOR Controller up and highly available, but do not have a container or VM platform in place to rely upon.  For situations like this we can easily leverage DRBD® and the Pacemaker/Corosync stack.

If familiar with Pacemaker, setting up a clustered LINSTOR Controller should seem pretty straightforward. The only really tricky bit here is that we first need to install LINSTOR to create the DRBD storage that will provide the storage for LINSTOR. Sounds a little bit chicken-and-egg, I know, but this allows LINSTOR to be aware of, and manage, all DRBD resources.

The below example is for only two nodes, but it could be easily adapted for more nodes. Make sure to install both the LINSTOR Controller and LINSTOR Satellite software to both nodes. The below instructions are by no means a step-by-step guide, but rather just the “special sauce” needed for a HA LINSTOR Controller cluster.

If using the LINBIT® provided package repositories an Ansible playbook is available to entirely automate the deployment of this cluster on a RHEL7 or CentOS7 system.

Create a DRBD resource for the LINSTOR database

We’ll name this resource linstordb, and use the already already configured pool0 storage pool.

[root@linstora ~]# linstor resource-definition create linstordb
[root@linstora ~]# linstor resource-definition drbd-options --on-no-quorum=io-error linstordb
[root@linstora ~]# linstor resource-definition drbd-options --auto-promote=no linstordb
[root@linstora ~]# linstor volume-definition create linstordb 250M
[root@linstora ~]# linstor resource create linstora linstordb --storage-pool pool0
[root@linstora ~]# linstor resource create linstorb linstordb --storage-pool pool0

Stop the LINSTOR Controller and move the database to the DRBD device

Move the database temporarily, mount the DRBD device where LINSTOR expects the database, and move it back. Make sure to execute the `chattr +i /var/lib/linstor` on all nodes that may potentially run a linstor-controller.

[root@linstora ~]# systemctl stop linstor-controller
[root@linstora ~]# rsync -avp /var/lib/linstor /tmp/
[root@linstora ~]# drbdadm primary linstordrb
[root@linstora ~]# mkfs.xfs /dev/drbd/by-res/linstordb/0
[root@linstora ~]# rm -rf /var/lib/linstor/*
[root@linstora ~]# chattr +i /var/lib/linstor  # only if on LINSTOR >= 1.14.0
[root@linstora ~]# mount /dev/drbd/by-res/linstordb/0 /var/lib/linstor
[root@linstora ~]# rsync -avp /tmp/linstor/ /var/lib/linstor/
[root@linstora ~]# drbdadm secondary linstordb
[root@linstora ~]# umount /dev/drbd/by-res/linstordb/0

Cluster everything up in Pacemaker

Please note that we strongly encourage you utilize tested and working STONITH in all Pacemaker cluster. This example omits it simply because these VMs did not have any fencing devices available.

primitive p_drbd_linstordb ocf:linbit:drbd \
        params drbd_resource=linstordb \
        op monitor interval=29 role=Master \
        op monitor interval=30 role=Slave \
        op start interval=0 timeout=240s \
        op stop interval=0 timeout=100s
primitive p_fs_linstordb Filesystem \
        params device="/dev/drbd/by-res/linstordb/0" directory="/var/lib/linstor" \
        op start interval=0 timeout=60s \
        op stop interval=0 timeout=100s \
        op monitor interval=20s timeout=40s
primitive p_linstor-controller systemd:linstor-controller \
        op start interval=0 timeout=100s \
        op stop interval=0 timeout=100s \
        op monitor interval=30s timeout=100s
ms ms_drbd_linstordb p_drbd_linstordb \
        meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
group g_linstor p_fs_linstordb p_linstor-controller
order o_drbd_before_linstor inf: ms_drbd_linstordb:promote g_linstor:start
colocation c_linstor_with_drbd inf: g_linstor ms_drbd_linstordb:Master
property cib-bootstrap-options: \
        stonith-enabled=false \
        no-quorum-policy=ignore

We still usually advise leveraging the features already built into your cloud or VM platform for high availability if one is available, but if not, you can always use the above to leverage pacemaker to make your LINSTOR Controller highly available.

Share on facebook
Share on twitter
Share on linkedin
Share on reddit
Share on whatsapp
Share on vk
Share on email

Share this post

Devin Vance

Devin Vance

First introduced to Linux back in 1996, and using Linux almost exclusively by 2005, Devin has years of Linux administration and systems engineering under his belt. He has been deploying and improving clusters with LINBIT since 2011. When not at the keyboard, you can usually find Devin wrenching on an American motorcycle or down at one of the local bowling alleys.

Talk to us

LINBIT is committed to protecting and respecting your privacy, and we’ll only use your personal information to administer your account and to provide the products and services you requested from us. From time to time, we would like to contact you about our products and services, as well as other content that may be of interest to you. If you consent to us contacting you for this purpose, please tick above to say how you would like us to contact you.

You can unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.

By clicking submit below, you consent to allow LINBIT to store and process the personal information submitted above to provide you the content requested.

Talk to us

LINBIT is committed to protecting and respecting your privacy, and we’ll only use your personal information to administer your account and to provide the products and services you requested from us. From time to time, we would like to contact you about our products and services, as well as other content that may be of interest to you. If you consent to us contacting you for this purpose, please tick above to say how you would like us to contact you.

You can unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.

By clicking submit below, you consent to allow LINBIT to store and process the personal information submitted above to provide you the content requested.