Part of the design of LINSTOR is that if the central LINSTOR Controller goes down, all the storage still remains up and accessible. This should allow ample time to repair the downed system hosting the LINSTOR Controller. Still, in the majority of cases, it is preferred to run the LINSTOR Controller in a container within your cloud or as a VM in your hypervisor platform. However, there may exist a situation where you want to keep the LINSTOR Controller up and highly available, but do not have a container or VM platform in place to rely upon. For situations like this we can easily leverage DRBD and the Pacemaker/Corosync stack.
If familiar with Pacemaker, setting up a clustered LINSTOR Controller should seem pretty straightforward. The only really tricky bit here is that we first need to install LINSTOR to create the DRBD storage that will provide the storage for LINSTOR. Sounds a little bit chicken-and-egg, I know, but this allows LINSTOR to be aware of, and manage, all DRBD resources.
The below example is for only two nodes, but it could be easily adapted for more nodes. Make sure to install both the LINSTOR Controller and LINSTOR Satellite software to both nodes. The below instructions are by no means a step-by-step guide, but rather just the “special sauce” needed for a HA LINSTOR Controller cluster.
If using the LINBIT provided package repositories an Ansible playbook is available to entirely automate the deployment of this cluster on a RHEL7 or CentOS7 system.
Create a DRBD resource for the LINSTOR database
We’ll name this resource linstordb, and use the already already configured pool0 storage pool.
[root@linstora ~]# linstor resource-definition create linstordb
[root@linstora ~]# linstor volume-definition create linstordb 250M
[root@linstora ~]# linstor resource create linstora linstordb --storage-pool pool0
[root@linstora ~]# linstor resource create linstorb linstordb --storage-pool pool0Stop the LINSTOR Controller and move the database to the DRBD device
Move the database temporarily, mount the DRBD device where LINSTOR expects the database, and move it back.
[root@linstora ~]# systemctl stop linstor-controller
[root@linstora ~]# rsync -avp /var/lib/linstor /tmp/
[root@linstora ~]# mkfs.xfs /dev/drbd/by-res/linstordb/0
[root@linstora ~]# rm -rf /var/lib/linstor/*
[root@linstora ~]# mount /dev/drbd/by-res/linstordb/0 /var/lib/linstor
[root@linstora ~]# rsync -avp /tmp/linstor/ /var/lib/linstor/
Cluster everything up in Pacemaker
Please note that we strongly encourage you utilize tested and working STONITH in all Pacemaker cluster. This example omits it simply because these VMs did not have any fencing devices available.
primitive p_drbd_linstordb ocf:linbit:drbd \
params drbd_resource=linstordb \
op monitor interval=29 role=Master \
op monitor interval=30 role=Slave \
op start interval=0 timeout=240s \
op stop interval=0 timeout=100s
primitive p_fs_linstordb Filesystem \
params device="/dev/drbd/by-res/linstordb/0" directory="/var/lib/linstor" \
op start interval=0 timeout=60s \
op stop interval=0 timeout=100s \
op monitor interval=20s timeout=40s
primitive p_linstor-controller systemd:linstor-controller \
op start interval=0 timeout=100s \
op stop interval=0 timeout=100s \
op monitor interval=30s timeout=100s
ms ms_drbd_linstordb p_drbd_linstordb \
meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
group g_linstor p_fs_linstordb p_linstor-controller
order o_drbd_before_linstor inf: ms_drbd_linstordb:promote g_linstor:start
colocation c_linstor_with_drbd inf: g_linstor ms_drbd_linstordb:Master
property cib-bootstrap-options: \
stonith-enabled=false \
no-quorum-policy=ignore
We still usually advise leveraging the features already built into your cloud or VM platform for high availability if one is available, but if not, you can always the above to leverage pacemaker to make your LINSTOR Controller highly available.
Photo credit to Gavin Whitner.
