How to Setup LINSTOR on Proxmox VE

Authors: Ryan Ronnander & Roland Kammerer

—

In this technical blog post, we show you how to integrate LINSTOR® with Proxmox VE using a storage plugin developed here at LINBIT®. The advantages of using LINSTOR with Proxmox VE include:

• Extremely fast live migrations of virtual machines (VMs) and containers.
• Highly available VMs and containers, ultimately mitigating virtual host failures.
• Full replicas of data (virtual disks) on every node.
• A configurable number of data replicas. For example, 3 replicas of a virtual disk in a 5-node cluster.

Setup Overview

The rest of this post assumes that you have already configured Proxmox VE (the example here uses three nodes), and have created a Proxmox VE cluster consisting of all three nodes. While this post is not meant to replace the relevant LINSTOR User’s guide sections, this post is a walk-through of the necessary steps to install and configure LINSTOR alongside Proxmox VE. There is also a companion video to complement this blog post.

The setup consists of two important components:

1. LINSTOR for managing and provisioning replicated storage.
2. The LINSTOR Proxmox plugin that provides the interface between Proxmox VE and LINSTOR.

LINBIT publishes public APT repositories for use with Proxmox VE. These repositories contain all required packages to create a fully functional LINSTOR cluster.

LINSTOR Cluster

First, you’ll need to set up the LINBIT Proxmox repository on all nodes.

Next, we need to install the low-level components for replication. This consists of the DRBD® 9 kernel module and the drbd-utils package.

Install the low-level components on all nodes:

apt install proxmox-default-headers drbd-dkms drbd-utils

Next, install LINSTOR and start the linstor-satellite service on all nodes:

apt install linstor-controller linstor-satellite linstor-client
systemctl enable linstor-satellite --now

Now, decide which of your hosts will be the current controller node and enable and start the linstor-controller service on that particular node only:

systemctl enable linstor-controller --now

On the node that runs the controller service, enter the following commands to add your nodes:

linstor node create proxmox-0 192.168.222.130
linstor node create proxmox-1 192.168.222.131
linstor node create proxmox-2 192.168.222.132

Configuring Storage

LINSTOR needs underlying storage to create replicated volumes. This post uses a configuration where all nodes contain an LVM thin pool called thinpool. In the demonstration environment, there is a thin pool created inside the linstor_vg volume group. In your environment, you will likely have a different storage topology.

In the demonstration environment, the following commands were run on all nodes to prepare the underlying storage for LINSTOR:

vgcreate linstor_vg /dev/vdb
lvcreate -l 80%FREE -T linstor_vg/thinpool

The next step is to configure a LINSTOR storage pool definition. Most LINSTOR objects consist of a “definition” and then concrete instances of such a definition. For a better understanding of these objects and their relationships, see our User’s Guide.

The next step is to create every node’s storage pool:

linstor storage-pool create lvmthin proxmox-0 pve-storage linstor_vg/thinpool
linstor storage-pool create lvmthin proxmox-1 pve-storage linstor_vg/thinpool
linstor storage-pool create lvmthin proxmox-2 pve-storage linstor_vg/thinpool

Now that you’ve run a few LINSTOR commands, it’s worth mentioning that the LINSTOR client provides handy shortcuts for its sub-commands. For example, to list all current LINSTOR nodes you can run linstor node list or the simplified linstor n l.

LINSTOR Resource Groups

Once your storage pool has been defined, you can create a LINSTOR resource group which will define how replicated volumes are provisioned in the cluster.

Next, create a resource group called pve-rg using the pve-storage storage pool:

linstor resource-group create pve-rg --storage-pool=pve-storage --place-count=3
linstor volume-group create pve-rg

Specifying --place-count=3 ensures each replicated volume will exist on all three nodes in the cluster. Each resource group also needs a corresponding volume group, which was created using the same name as the resource group.

LINSTOR Proxmox VE Plugin Setup

As DRBD and LINSTOR are already configured, the only thing left to do is install the linstor-proxmox plugin package and configure the plugin:

apt install linstor-proxmox

The plugin is configured through the Proxmox storage configuration file located at /etc/pve/storage.cfg. Add the following section on one node:

drbd: linstor_storage
    content images, rootdir
    controller 192.168.222.130
    resourcegroup pve-rg

It is not necessary to copy storage.cfg to the other nodes as /etc/pve is a replicated file system in Proxmox VE.

After the configuration is done, restart the following services on each node:

systemctl restart pve-cluster pvedaemon pvestatd pveproxy pve-ha-lrm

Now you are able to create VMs and containers backed by LINSTOR from the GUI. To do so, simply select linstor_storage as the storage in the “Disks” section of the VM configuration. LINSTOR then provisions the replicated storage based on the resource group settings defined earlier in this post.

The status of replicated storage in the cluster can be viewed using the linstor resource list command.

Diskless Resources

To highlight a benefit of using LINSTOR, assume that you initially created five Proxmox VE nodes in the cluster, but only three nodes have physical backing storage. It is important to know that the storage can be accessed by all nodes in the cluster with a feature called “diskless clients”.

Now, let us migrate a VM to a new node without any backing storage (proxmox-3). This is again done with a few clicks in the GUI, but the interesting steps happen behind the scenes: The storage plugin realizes that it has access to the data on proxmox-0, proxmox-1, and proxmox-2 (your three replicas), but also needs access on proxmox-3 to run the VM. The plugin then creates a diskless assignment on proxmox-3. When you execute linstor resource list, you can see four nodes are now involved:

• proxmox-0 with storage in Secondary (Unused) role.
• proxmox-1 with storage in Secondary (Unused) role.
• proxmox-2 with storage in Secondary (Unused) role.
• proxmox-3 as a diskless client in Primary (InUse) role.

When moving VMs around inside the GUI, diskless clients are created (and deleted) on demand without further user interaction. This means that if you now move the VM back to a node with storage, the diskless assignment gets deleted as it is no longer needed.

This flexibility allows LINSTOR to be used across many different storage topologies and server configurations.

Next Steps

Up to this point you have created highly available replicated storage for your VMs and containers, however, the LINSTOR controller and especially its database itself are not highly available. For production deployments consider making the LINSTOR controller service highly available in your cluster.

With three or more Proxmox VE nodes and LINSTOR functioning as “shared” storage in your Proxmox cluster, you can now make your VMs and containers highly available. Again, you can view the companion video walk-through to see just how easy it is to configure LINSTOR for use with Proxmox VE.

If you’re interested in obtaining support for LINSTOR, don’t hesitate to reach out to us for more information.