Authors: Ryan Ronnander & Roland Kammerer
In this technical blog post, we show you how to integrate LINSTOR® with Proxmox VE using a storage plugin developed here at LINBIT®. The advantages of using LINSTOR with Proxmox VE include:
- Extremely fast live migrations of virtual machines (VMs) and containers.
- Highly available VMs and containers, ultimately mitigating virtual host failures.
- Full replicas of data (virtual disks) on every node.
- A configurable number of data replicas. For example, 3 replicas of a virtual disk in a 5-node cluster.
The rest of this post assumes that you have already configured Proxmox VE (the example here uses three nodes), and have created a Proxmox VE cluster consisting of all three nodes. While this post is not meant to replace the relevant LINSTOR User’s guide sections, this post is a walk-through of the necessary steps to install and configure LINSTOR alongside Proxmox VE. There is also a companion video to complement this blog post.
The setup consists of two important components:
- LINSTOR for managing and provisioning replicated storage.
- The LINSTOR Proxmox plugin that provides the interface between Proxmox VE and LINSTOR.
LINBIT publishes public APT repositories for use with Proxmox VE. These repositories contain all required packages to create a fully functional LINSTOR cluster.
First, you’ll need to set up the LINBIT Proxmox repository on all nodes.
Next, we need to install the low-level components for replication. This consists of the DRBD® 9 kernel module and the
drbd-utils package. Before blindly installing the
pve-headers package, we recommend running
uname -a to ensure you’re installing matching kernel headers for your currently installed Linux kernel.
Install the low-level components on all nodes:
apt install pve-headers drbd-dkms drbd-utils
Next, install LINSTOR and start the
linstor-satellite service on all nodes:
apt install linstor-controller linstor-satellite linstor-client systemctl enable linstor-satellite --now
Now, decide which of your hosts will be the current controller node and enable and start the
linstor-controller service on that particular node only:
systemctl enable linstor-controller --now
On the node that runs the controller service, enter the following commands to add your nodes:
linstor node create proxmox-0 192.168.222.130 linstor node create proxmox-1 192.168.222.131 linstor node create proxmox-2 192.168.222.132
LINSTOR needs underlying storage to create replicated volumes. This post uses a configuration where all nodes contain an LVM thin pool called
thinpool. In the demonstration environment, there is a thin pool created inside the
linstor_vg volume group. In your environment, you will likely have a different storage topology.
In the demonstration environment, the following commands were run on all nodes to prepare the underlying storage for LINSTOR:
vgcreate linstor_vg /dev/vdb lvcreate -l 80%FREE -T linstor_vg/thinpool
The next step is to configure a LINSTOR storage pool definition. Most LINSTOR objects consist of a “definition” and then concrete instances of such a definition. For a better understanding of these objects and their relationships, see our User’s Guide.
The next step is to create every node’s storage pool:
linstor storage-pool create lvmthin proxmox-0 pve-storage linstor_vg/thinpool linstor storage-pool create lvmthin proxmox-1 pve-storage linstor_vg/thinpool linstor storage-pool create lvmthin proxmox-2 pve-storage linstor_vg/thinpool
Now that you’ve run a few LINSTOR commands, it’s worth mentioning that the LINSTOR client provides handy shortcuts for its sub-commands. For example, to list all current LINSTOR nodes you can run
linstor node list or the simplified
linstor n l.
LINSTOR Resource Groups
Once your storage pool has been defined, you can create a LINSTOR resource group which will define how replicated volumes are provisioned in the cluster.
Next, create a resource group called
pve-rg using the
pve-storage storage pool:
linstor resource-group create pve-rg --storage-pool=pve-storage --place-count=3 linstor volume-group create pve-rg
--place-count=3 ensures each replicated volume will exist on all three nodes in the cluster. Each resource group also needs a corresponding volume group, which was created using the same name as the resource group.
LINSTOR Proxmox VE Plugin Setup
As DRBD and LINSTOR are already configured, the only thing left to do is install the
linstor-proxmox plugin package and configure the plugin:
apt install linstor-proxmox
The plugin is configured through the Proxmox storage configuration file located at
/etc/pve/storage.cfg. Add the following section on one node:
drbd: linstor_storage content images, rootdir controller 192.168.222.130 resourcegroup pve-rg
It is not necessary to copy
storage.cfg to the other nodes as
/etc/pve is a replicated file system in Proxmox VE.
After the configuration is done, restart the following services on each node:
systemctl restart pve-cluster pvedaemon pvestatd pveproxy pve-ha-lrm
Now you are able to create VMs and containers backed by LINSTOR from the GUI. To do so, simply select
linstor_storage as the storage in the “Disks” section of the VM configuration. LINSTOR then provisions the replicated storage based on the resource group settings defined earlier in this post.
The status of replicated storage in the cluster can be viewed using the
linstor resource list command.
To highlight a benefit of using LINSTOR, assume that you initially created five Proxmox VE nodes in the cluster, but only three nodes have physical backing storage. It is important to know that the storage can be accessed by all nodes in the cluster with a feature called “diskless clients”.
Now, let us migrate a VM to a new node without any backing storage (
proxmox-3). This is again done with a few clicks in the GUI, but the interesting steps happen behind the scenes: The storage plugin realizes that it has access to the data on
proxmox-2 (your three replicas), but also needs access on
proxmox-3 to run the VM. The plugin then creates a diskless assignment on
proxmox-3. When you execute
linstor resource list, you can see four nodes are now involved:
proxmox-0with storage in Secondary (Unused) role.
proxmox-1with storage in Secondary (Unused) role.
proxmox-2with storage in Secondary (Unused) role.
proxmox-3as a diskless client in Primary (InUse) role.
When moving VMs around inside the GUI, diskless clients are created (and deleted) on demand without further user interaction. This means that if you now move the VM back to a node with storage, the diskless assignment gets deleted as it is no longer needed.
This flexibility allows LINSTOR to be used across many different storage topologies and server configurations.
Up to this point you have created highly available replicated storage for your VMs and containers, however, the LINSTOR controller and especially its database itself are not highly available. For production deployments consider making the LINSTOR controller service highly available in your cluster.
With three or more Proxmox VE nodes and LINSTOR functioning as “shared” storage in your Proxmox cluster, you can now make your VMs and containers highly available. Again, you can view the companion video walk-through to see just how easy it is to configure LINSTOR for use with Proxmox VE.
If you’re interested in obtaining support for LINSTOR, don’t hesitate to reach out to us for more information.