Multi-AZ Replication Using Automatic Placement Rules in LINSTOR

Multi-AZ Replication

Multi-AZ Replication

Are you looking to leverage multiple availability zones in Amazon Web Services (AWS) or Microsoft Azure? LINSTOR can easily manage your block storage and automatically replicate your data between availability zones (AZs). This post explores multi-AZ replication using automatic placement rules in LINSTOR.

In this blog, we will take advantage of LINSTOR’s lesser-known auxiliary properties. With just a few simple configuration changes, we can ensure newly created resources always have data replicas deployed across multiple availability zones (AZs).

LINSTOR Cluster Configuration

The LINSTOR cluster used in this blog will have the following characteristics:

  • Five LINSTOR nodes (total) deployed within three AZs.
  • LINSTOR node pairs (four total) deployed across two primary AZs: AZ-1 and AZ-2.
  • The four nodes in AZ-1 and AZ-2 have all been configured with the sp0 storage pool for replica storage.
  • A single LINSTOR controller node in AZ-3. This node does not contain any replica storage and will also act as a tiebreaker node for quorum purposes.

LINSTOR cluster node list:

[root@linstor-ctrl-0 ~]# linstor node list

╭────────────────────────────────────────────────────────────────────╮
┊ Node           ┊ NodeType  ┊ Addresses                    ┊ State  ┊
╞════════════════════════════════════════════════════════════════════╡
┊ linstor-ctrl-0 ┊ COMBINED  ┊ 192.168.222.250:3376 (PLAIN) ┊ Online ┊
┊ linstor-sat-0  ┊ SATELLITE ┊ 192.168.222.10:3366 (PLAIN)  ┊ Online ┊
┊ linstor-sat-1  ┊ SATELLITE ┊ 192.168.222.11:3366 (PLAIN)  ┊ Online ┊
┊ linstor-sat-2  ┊ SATELLITE ┊ 192.168.222.12:3366 (PLAIN)  ┊ Online ┊
┊ linstor-sat-3  ┊ SATELLITE ┊ 192.168.222.13:3366 (PLAIN)  ┊ Online ┊
╰────────────────────────────────────────────────────────────────────╯

Note the LINSTOR controller node is a COMBINED type, functioning both as a controller node and a satellite node.

LINSTOR cluster storage pool listing:

[root@linstor-ctrl-0 ~]# linstor storage-pool list

╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node           ┊ Driver   ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ linstor-ctrl-0 ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ DfltDisklessStorPool ┊ linstor-sat-0  ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ DfltDisklessStorPool ┊ linstor-sat-1  ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ DfltDisklessStorPool ┊ linstor-sat-2  ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ DfltDisklessStorPool ┊ linstor-sat-3  ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ sp0                  ┊ linstor-sat-0  ┊ LVM      ┊ drbdpool ┊     3.74 GiB ┊      8.00 GiB ┊ False        ┊ Ok    ┊            ┊
┊ sp0                  ┊ linstor-sat-1  ┊ LVM      ┊ drbdpool ┊     3.74 GiB ┊      8.00 GiB ┊ False        ┊ Ok    ┊            ┊
┊ sp0                  ┊ linstor-sat-2  ┊ LVM      ┊ drbdpool ┊     3.74 GiB ┊      8.00 GiB ┊ False        ┊ Ok    ┊            ┊
┊ sp0                  ┊ linstor-sat-3  ┊ LVM      ┊ drbdpool ┊     3.74 GiB ┊      8.00 GiB ┊ False        ┊ Ok    ┊            ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Spawning Resources Without Auxiliary Properties

In LINSTOR clusters with three or more satellite nodes, issuing a spawn-resources command by default creates a new resource with two diskful replicas and one diskless tiebreaker. However, nothing guarantees or specifies where the replicas or tiebreaker are placed.

With multiple LINSTOR nodes residing in both AZ-1 and AZ-2, issuing a spawn-resources command after the initial cluster deployment results in randomly placed replicas. Each replica may end up spread across two nodes in the same AZ. Other times, replicas are spread across separate AZs.

The amount of nodes illuminates the problem in this case. For example, if we had a three-node LINSTOR cluster deployed across three AZs, there would always be a replica in different AZs due to the node count.

Defining Auxiliary Properties for Availability Zones

We can assign an AZ to each node. We will use auxiliary properties to take the first step in configuring our cluster for improved replica assignment:

# set availability_zone auxiliary properties for each node
linstor node set-property –aux linstor-sat-0 availability_zone AZ-1
linstor node set-property –aux linstor-sat-1 availability_zone AZ-1
linstor node set-property –aux linstor-sat-2 availability_zone AZ-2
linstor node set-property –aux linstor-sat-3 availability_zone AZ-2
linstor node set-property –aux linstor-ctrl-0 availability_zone AZ-3 # quorum node

New auxiliary properties listed for each node:

[root@linstor-ctrl-0 ~]# linstor node list –show-aux-props

╭─────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ Node           ┊ NodeType  ┊ Addresses                    ┊ AuxProps                   ┊ State  ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ linstor-ctrl-0 ┊ COMBINED  ┊ 192.168.222.250:3366 (PLAIN) ┊ Aux/availability_zone=AZ-3 ┊ Online ┊
┊ linstor-sat-0  ┊ SATELLITE ┊ 192.168.222.10:3366 (PLAIN)  ┊ Aux/availability_zone=AZ-1 ┊ Online ┊
┊ linstor-sat-1  ┊ SATELLITE ┊ 192.168.222.11:3366 (PLAIN)  ┊ Aux/availability_zone=AZ-1 ┊ Online ┊
┊ linstor-sat-2  ┊ SATELLITE ┊ 192.168.222.12:3366 (PLAIN)  ┊ Aux/availability_zone=AZ-2 ┊ Online ┊
┊ linstor-sat-3  ┊ SATELLITE ┊ 192.168.222.13:3366 (PLAIN)  ┊ Aux/availability_zone=AZ-2 ┊ Online ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────╯

Creating the Resource Group

Now that each LINSTOR node’s auxiliary properties are properly configured, we can leverage the new properties in our resource group definitions.

Create a new resource group named rg0 on top of the sp0 storage pool:

# rg0 resource group defined with two replicas across different AZs
linstor resource-group create –storage-pool sp0 –place-count 2 rg0 –replicas-on-different availability_zone

# create the corresponding volume group
linstor volume-group create rg0

Take note of the –replicas-on-different parameter above, which allows us to specify that replicas will be placed on different AZs. Existing resource groups can also be updated with new parameters if you’re using an already configured LINSTOR cluster.

The new configuration for rg0:

[root@linstor-ctrl-0 ~]# linstor resource-group list –resource-groups rg0

╭──────────────────────────────────────────────────────────────────────╮
┊ ResourceGroup ┊ SelectFilter                  ┊ VlmNrs ┊ Description ┊
╞══════════════════════════════════════════════════════════════════════╡
┊ rg0           ┊ PlaceCount: 2                 ┊ 0      ┊             ┊
┊               ┊ StoragePool(s): sp0           ┊        ┊             ┊
┊               ┊ ReplicasOnDifferent:          ┊        ┊             ┊
┊               ┊  ['Aux/availability_zone']    ┊        ┊             ┊
╰──────────────────────────────────────────────────────────────────────╯

Spawning Resources

Now that our configuration is finalized, we can create some new resources in rg0 using spawn-resources:

# spawn various resources for intended use with MySQL databases
linstor resource-group spawn-resources rg0 mysql_001 256M
linstor resource-group spawn-resources rg0 mysql_002 256M

After creating these new resources, let’s see how they’re distributed throughout our LINSTOR cluster:

[root@linstor-ctrl-0 ~]# linstor volume list

╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ Node           ┊ Resource  ┊ StoragePool          ┊ VolNr ┊ MinorNr ┊ DeviceName    ┊ Allocated ┊ InUse  ┊      State ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ linstor-ctrl-0 ┊ mysql_001 ┊ DfltDisklessStorPool ┊     0 ┊    1000 ┊ /dev/drbd1000 ┊           ┊ Unused ┊ TieBreaker ┊
┊ linstor-sat-0  ┊ mysql_001 ┊ sp0                  ┊     0 ┊    1000 ┊ /dev/drbd1000 ┊   260 MiB ┊ Unused ┊   UpToDate ┊
┊ linstor-sat-2  ┊ mysql_001 ┊ sp0                  ┊     0 ┊    1000 ┊ /dev/drbd1000 ┊   260 MiB ┊ Unused ┊   UpToDate ┊
┊ linstor-ctrl-0 ┊ mysql_002 ┊ DfltDisklessStorPool ┊     0 ┊    1001 ┊ /dev/drbd1001 ┊           ┊ Unused ┊ TieBreaker ┊
┊ linstor-sat-1  ┊ mysql_002 ┊ sp0                  ┊     0 ┊    1001 ┊ /dev/drbd1001 ┊   260 MiB ┊ Unused ┊   UpToDate ┊
┊ linstor-sat-3  ┊ mysql_002 ┊ sp0                  ┊     0 ┊    1001 ┊ /dev/drbd1001 ┊   260 MiB ┊ Unused ┊   UpToDate ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Looking through the above list of newly created DRBD volumes in the cluster, we can see that the mysql_001 resource has been created with replicas on linstor-sat-0 (AZ-1) as well as linstor-sat-2 (AZ-2). Also, the mysql_002 resource has been created with replicas on linstor-sat-1 (AZ-1) and linstor-sat-3 (AZ-2). Both resources were created with their respective replicas placed inside different AZs (AZ-1 and AZ-2) as specified in the resource group.

Conclusion

After minor changes to a typical LINSTOR deployment, LINSTOR resources are now easily created with replicas that permanently reside within different AZs.

Replicating between multiple regions is also possible by extending LINSTOR with our closed source DRBD Proxy solution. Due to the higher latency between regions, DRBD Proxy becomes a necessary add-on, enabling asynchronous replication between regions without impacting performance. LINSTOR, DRBD, and DRBD Proxy are all components of LINBIT’s Software Defined Storage (SDS) solution.

To learn more about how LINSTOR can simplify your block storage management and data replication requirements in a Multi-AZ or Multi-Region deployment, reach out to us and request a support evaluation today.

Share on facebook
Share on twitter
Share on linkedin
Share on reddit
Share on whatsapp
Share on vk
Share on email

Share this post

Talk to us

LINBIT is committed to protecting and respecting your privacy, and we’ll only use your personal information to administer your account and to provide the products and services you requested from us. From time to time, we would like to contact you about our products and services, as well as other content that may be of interest to you. If you consent to us contacting you for this purpose, please tick above to say how you would like us to contact you.

You can unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.

By clicking submit below, you consent to allow LINBIT to store and process the personal information submitted above to provide you the content requested.

Talk to us

LINBIT is committed to protecting and respecting your privacy, and we’ll only use your personal information to administer your account and to provide the products and services you requested from us. From time to time, we would like to contact you about our products and services, as well as other content that may be of interest to you. If you consent to us contacting you for this purpose, please tick above to say how you would like us to contact you.

You can unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.

By clicking submit below, you consent to allow LINBIT to store and process the personal information submitted above to provide you the content requested.