LINSTOR用户指南

请先看这个

This guide is intended to serve users of the software-defined storage (SDS) solution LINSTOR® as a definitive reference guide and handbook.

本指南始终假设您正在使用最新版本的LINSTOR和相关工具。

本指南的组织如下:

LINSTOR

1. Basic Administrative Tasks and System Setup

LINSTOR is a configuration management system for storage on Linux systems. It manages LVM logical volumes, ZFS ZVOLs, or both, on a cluster of nodes. It leverages DRBD for replication between different nodes and to provide block storage devices to users and applications. It manages snapshots, encryption and caching of HDD backed data in SSDs using bcache.

1.1. 概念和术语

本节将介绍一些核心概念和术语,您需要熟悉这些概念和术语才能理解LINSTOR是如何工作和部署存储的。这一部分是以 “ground up” 的方式布置的。

1.1.1. 可安装组件

linstor-controller

LINSTOR设置至少需要一个活动的controller和一个或多个satellites。

The linstor-controller relies on a database that holds all configuration information for the whole cluster. It makes all decisions that need to have a view of the whole cluster. Multiple controllers can be used for LINSTOR but only one can be active.

linstor-satellite

linstor-satellite 运行在每个LINSTOR使用本地存储或为服务提供存储的节点上。它是无状态的;它从控制器接收所需的所有信息。它运行 lvcreatedrbdadm 等程序。它行为方式就像一个节点代理。

linstor-client

linstor-client 是一个命令行实用程序,用于向系统发送命令并检查系统的状态。

1.1.2. 对象

Objects are the end result which LINSTOR presents to the end-user or application, such as: Kubernetes/OpenShift, a replicated block device (DRBD), NVMeOF target, and others.

节点

Nodes are a server or container that participate in a LINSTOR cluster. The Node attribute defines:

  • 确定节点参与哪个LINSTOR集群

  • 设置节点的角色:Controller, Satellite, Auxiliary

  • 网络接口 对象定义了节点的网络连接

网络接口

顾名思义,这就是定义节点网络接口的接口/地址的方式。

Definitions

Definitions define attributes of an object, and can be thought of as profiles or templates. Objects created will inherit the configuration defined in the definitions. A definition must be defined prior to creating the associated object. For example, you must create a ResourceDefinition prior to creating the Resource.

StoragePoolDefinition
  • 定义存储池的名称

ResourceDefinition

资源 definitions定义资源的以下属性:

  • DRBD资源的名称

  • 用于资源连接的DRBD的TCP端口

VolumeDefinition

Volume definitions 定义如下:

  • DRBD资源的卷

  • 卷的大小

  • DRBD资源卷的卷号

  • 卷的元数据属性

  • 用于与DRBD卷关联的DRBD设备的次要编号

StoragePool

StoragePool 用于标识LINSTOR上下文中的存储。它定义了:

  • 特定节点上存储池的配置

  • The storage back-end driver to use for the storage pool on the cluster node (LVM, ZFS, and others)

  • 要传递给存储备份驱动程序的参数和配置

Resource

LINSTOR has now expanded its capabilities to manage a broader set of storage technologies outside of just DRBD. A Resource:

  • 表示在 ResourceDefinition 中定义的DRBD资源的位置

  • 在群集中的节点上放置资源

  • 定义节点上 ResourceDefinition 的位置

Volume

Volumes are a subset of a Resource. A Resource could have multiple volumes, for example you may want to have your database stored on slower storage than your logs in your MySQL cluster.

By keeping the volumes under a single resource you are essentially creating a consistency group. The Volume attribute can define also define attributes on a more granular level.

1.2. 更广泛的背景

虽然LINSTOR可以用来使DRBD的管理更加方便,但它通常与更高层次的软件栈集成。这种集成已经存在于Kubernetes、OpenStack、OpenNebula和Proxmox中。本指南中包含了在这些环境中部署LINSTOR的特定章节。

LINSTOR使用的南向驱动程序是LVM、thinLVM和ZFS。

1.3. 包

LINSTOR is packaged in both the RPM and the DEB variants:

  1. linstor-client 包含命令行客户端程序。这取决于通常已经安装的python。在RHEL8系统中,需要创建python符号链接

  2. linstor-controllerlinstor-satellite 都包含服务的系统单元文件。它们依赖于Java runtime environment(JRE)1.8版(headless)或更高版本。

有关这些软件包的更多详细信息,请参见上面的Installable Components部分。

If you have a support subscription to LINBIT®, you will have access to our certified binaries through our repositories.

1.4. FIPS Compliance

“This standard shall be used in designing and implementing cryptographic modules…​” — NIST’s FIPS 140-3 publication

You can configure LINSTOR to encrypt storage volumes, by using LUKS (dm-crypt), as detailed in the Encrypted Volumes section of this user’s guide. Refer to the LUKS and dm-crypt projects for FIPS compliance status.

You can also configure LINSTOR to encrypt communication traffic between a LINSTOR satellite and a LINSTOR controller, by using SSL/TLS, as detailed in the Secure Satellite Connections section of this user’s guide.

LINSTOR can also interface with Self-Encrypting Drives (SEDs) and you can use LINSTOR to initialize an SED drive. LINSTOR stores the drive’s password as a property that applies to the storage pool associated with the drive. LINSTOR encrypts the SED drive password by using the LINSTOR master passphrase that you must create first.

By default, LINSTOR uses the following cryptographic algorithms:

  • HMAC-SHA2-512

  • PBKDF2

  • AES-128

A FIPS compliant version of LINSTOR is available for the use cases mentioned in this section. If you or your organization require FIPS compliance at this level, please contact [email protected] for details.

1.5. Installing LINSTOR

If you want to use LINSTOR in containers, skip this section and use the Containers section below for the installation.

1.5.1. Installing a Volume Manager

To use LINSTOR to create storage volumes, you will need to install a volume manager, either LVM or ZFS, if one is not already installed on your system.

1.5.2. Using a Script to Manage LINBIT Cluster Nodes

If you are a LINBIT customer, you can download a LINBIT created helper script and run it on your nodes to:

  • Register a cluster node with LINBIT.

  • Join a node to an existing LINBIT cluster.

  • Enable LINBIT package repositories on your node.

Enabling LINBIT package repositories will give you access to LINBT software packages, DRBD kernel modules, and other related software such as cluster managers and OCF scripts. You can then use a package manager to fetch, install, and manage updating installed packages.

Downloading the LINBIT Manage Nodes Script

To register your cluster nodes with LINBIT, and configure LINBIT’s repositories, first download and then run the manage nodes helper script by entering the following commands on all cluster nodes:

# curl -O https://my.linbit.com/linbit-manage-node.py
# chmod +x ./linbit-manage-node.py
# ./linbit-manage-node.py
You must run the script as the root user.

The script will prompt you for your LINBIT customer portal username and password. After entering your credentials, the script will list cluster nodes associated with your account (none at first).

Enabling LINBIT Package Repositories

After you specify which cluster to register the node with, have the script write the registration data to a JSON file when prompted. Next, the script will show you a list of LINBIT repositories that you can enable or disable. You can find LINSTOR and other related packages in the drbd-9 repository. In most cases, unless you have a need to be on a different DRBD version branch, you should enable at least this repository.

Final Tasks Within Manage Nodes Script

After you have finished making your repositories selection, you can write the configuration to a file by following the script’s prompting. Next, be sure to answer yes to the question about installing LINBIT’s public signing key to your node’s keyring.

Before it closes, the script will show a message that suggests different packages that you can install for different use cases.

On DEB based systems you can install a precompiled DRBD kernel module package, drbd-module-$(uname -r), or a source version of the kernel module, drbd-dkms. Install one or the other package but not both.

1.5.3. Using a Package Manager to Install LINSTOR

After registering your node and enabling the drbd-9 LINBIT package repository, you can use a DEB, RPM, or YaST2 based package manager to install LINSTOR and related components.

If you are using a DEB based package manager, refresh your package repositories list by entering: apt update, before proceeding.
Installing DRBD Packages for Replicated LINSTOR Storage
If you will be using LINSTOR without DRBD, you can skip installing theses packages.

If you want to be able to use LINSTOR to create DRBD replicated storage, you will need to install the required DRBD packages. Depending on the Linux distribution that you are running on your node, install the DRBD-related packages that the helper script suggested. If you need to review the script’s suggested packages and installation commands, you can enter:

# ./linbit-manage-node.py --hints
Installing LINSTOR Packages

To install LINSTOR on a controller node, use your package manager to install the linbit-sds-controller package.

To install LINSTOR on a satellite node, use your package manager to install the linbit-sds-satellite package.

Install both packages if your node will be both a satellite and controller (Combined role).

1.5.4. Installing LINSTOR from Source Code

The LINSTOR project’s GitHub page is here: https://github.com/LINBIT/linstor-server.

LINBIT also has downloadable archived files of source code for LINSTOR, DRBD, and more, available here: https://linbit.com/linbit-software-download-page-for-linstor-and-drbd-linux-driver/.

1.6. Upgrading LINSTOR

LINSTOR doesn’t support rolling upgrades. Controller and satellites must have the same version, otherwise the controller will discard the satellite with a VERSION_MISMATCH. But this isn’t a problem, as the satellite won’t do any actions as long it isn’t connected to a controller and DRBD will not be disrupted by any means.

If you are using the embedded default H2 database and the linstor-controller package is upgraded an automatic backup file of the database will be created in the default /var/lib/linstor directory. This file is a good restore point if for any reason a linstor-controller database migration should fail, then it is recommended to report the error to LINBIT and restore the old database file and roll back to your previous controller version.

如果使用任何外部数据库或etcd,建议手动备份当前数据库以获得还原点。

First, upgrade the linstor-controller and linstor-client packages on your controller host and then restart the linstor-controller. The controller should start and all of its clients should show OFFLINE(VERSION_MISMATCH). After that you can continue upgrading the linstor-satellite package on all satellite nodes and restart them, after a short reconnection time they should all show ONLINE again and your upgrade is finished.

1.7. 容器

LINSTOR and related software are also available as containers. The base images are available in LINBIT’s container registry, drbd.io.

LINBIT’s container image repository (http://drbd.io) is only available to LINBIT customers or through LINBIT customer trial accounts. Contact LINBIT for information on pricing or to begin a trial. Alternatively, you may use LINSTOR SDS’ upstream project named Piraeus, without being a LINBIT customer.

To access the images, you first have to login to the registry using your LINBIT Customer Portal credentials.

# docker login drbd.io

The containers available in this repository are:

  • drbd.io/drbd9-rhel8

  • drbd.io/drbd9-rhel7

  • drbd.io/drbd9-sles15sp1

  • drbd.io/drbd9-bionic

  • drbd.io/drbd9-focal

  • drbd.io/linstor-csi

  • drbd.io/linstor-controller

  • drbd.io/linstor-satellite

  • drbd.io/linstor-client

An up-to-date list of available images with versions can be retrieved by opening http://drbd.io in your browser. Be sure to access the image repository through “http”, although the registry’s images themselves are pulled through “https”, using the associated docker pull command.

To load the kernel module, needed only for LINSTOR satellites, you’ll need to run a drbd9-$dist container in privileged mode. The kernel module containers either retrieve an official LINBIT package from a customer repository, use shipped packages, or they try to build the kernel modules from source. If you intend to build from source, you need to have the according kernel headers (e.g., kernel-devel) installed on the host. There are 4 ways to execute such a module load container:

  • Building from shipped source

  • Using a shipped/pre-built kernel module

  • 指定LINBIT节点哈希和容器版本。

  • 绑定装入现有仓库配置。

从已分发代码生成的示例(基于RHEL):

# docker run -it --rm --privileged -v /lib/modules:/lib/modules \
  -v /usr/src:/usr/src:ro \
  drbd.io/drbd9-rhel7

Example using a module shipped with the container, which is enabled by not bind-mounting /usr/src:

# docker run -it --rm --privileged -v /lib/modules:/lib/modules \
  drbd.io/drbd9-rhel8

Example using a hash and a distribution (rarely used):

# docker run -it --rm --privileged -v /lib/modules:/lib/modules \
  -e LB_DIST=rhel7.7 -e LB_HASH=ThisIsMyNodeHash \
  drbd.io/drbd9-rhel7

Example using an existing repo configuration (rarely used):

# docker run -it --rm --privileged -v /lib/modules:/lib/modules \
  -v /etc/yum.repos.d/linbit.repo:/etc/yum.repos.d/linbit.repo:ro \
  drbd.io/drbd9-rhel7
In both cases (hash + distribution, as well as bind-mounting a repo) the hash or repo configuration has to be from a node that has a special property set. Feel free to contact our support, and we will set this property.
For now (i.e., pre DRBD 9 version “9.0.17”), you must use the containerized DRBD kernel module, as opposed to loading a kernel module onto the host system. If you intend to use the containers you should not install the DRBD kernel module on your host systems. For DRBD version 9.0.17 or greater, you can install the kernel module as usual on the host system, but you need to load the module with the usermode_helper=disabled parameter (e.g., modprobe drbd usermode_helper=disabled).

然后以守护进程的身份运行LINSTOR satellite容器, 也需具有特权:

# docker run -d --name=linstor-satellite --net=host -v /dev:/dev \
  --privileged drbd.io/linstor-satellite
net=host is required for the containerized drbd-utils to be able to communicate with the host-kernel through Netlink.

To run the LINSTOR controller container as a daemon, mapping TCP port 3370 on the host to the container, enter the following command:

# docker run -d --name=linstor-controller -p 3370:3370 drbd.io/linstor-controller

To interact with the containerized LINSTOR cluster, you can either use a LINSTOR client installed on a system using repository packages, or using the containerized LINSTOR client. To use the LINSTOR client container:

# docker run -it --rm -e LS_CONTROLLERS=<controller-host-IP-address> \
  drbd.io/linstor-client node list

从这里开始,您可以使用LINSTOR客户端初始化集群,并开始使用典型的LINSTOR模式创建资源。

要停止并删除守护的容器和映像,请执行以下操作:

# docker stop linstor-controller
# docker rm linstor-controller

1.8. Initializing Your Cluster

我们假设在 所有 集群节点上已完成以下步骤:

  1. The DRBD9 kernel module is installed and loaded.

  2. drbd-utils are installed.

  3. LVM tools are installed.

  4. linstor-controller and/or linstor-satellite its dependencies are installed.

  5. The linstor-client is installed on the linstor-controller node.

Enable and also start the linstor-controller service on the host where it has been installed:

# systemctl enable --now linstor-controller

1.9. Using the LINSTOR Client

Whenever you run the LINSTOR command line client, it needs to know where your linstor-controller runs. If you do not specify it, it will try to reach a locally running linstor-controller listening on IP 127.0.0.1 port 3370. Therefore we will use the linstor-client on the same host as the linstor-controller.

The linstor-satellite requires TCP ports 3366 and 3367. The linstor-controller requires TCP port 3370. Verify that you have this port allowed on your firewall.
# linstor node list

Output from this command should show you an empty list and not an error message.

您可以在任何其他机器上使用 linstor 命令,但随后您需要告诉客户端如何找到linstor-controller。如下所示,可以将其指定为命令行选项、环境变量或全局文件:

# linstor --controllers=alice node list
# LS_CONTROLLERS=alice linstor node list

或者,您可以创建 /etc/linstor/linstor-client.conf 文件,并按如下方式填充它。

[global]
controllers=alice

如果配置了多个linstor控制器,只需在逗号分隔的列表中指定它们即可。linstor客户机会按照列出的顺序进行尝试。

linstor-client命令也可以通过只写参数的起始字母来快速方便地使用,例如: linstor node listlinstor n l

1.10. Adding Nodes to Your Cluster

The next step is to add nodes to your LINSTOR cluster.

# linstor node create bravo 10.43.70.3

If the IP is omitted, the client will try to resolve the given node-name as host-name by itself.

LINSTOR will automatically detect the node’s local uname -n which is later used for the DRBD-resource.

使用 `linstor node list`时,将看到新节点标记为脱机。现在启动并启用该节点上的 linstor-satellite,以便服务在重新启动时也启动:

# systemctl enable --now  linstor-satellite

如果您确定该服务已默认启用并在重新启动时启动,则还可以使用 systemctl start linstor-satellite

大约10秒后,您将看到 linstor node list 中的状态变为联机。当然,在controller知道satellite节点的存在之前,satellite进程已经启动完毕。

如果承载controller的节点也应该为LINSTOR集群提供存储空间,则必须将其添加为节点并启动linstor-satellite。

If you want to have other services wait until the linstor-satellite had a chance to create the necessary devices (that is, after a boot), you can update the corresponding .service file and change Type=simple to Type=notify.

This will cause the satellite to delay sending the READY=1 message to systemd until the controller connects, sends all required data to the satellite and the satellite at least tried once to get the devices up and running.

1.11. Storage Pools

StoragePools在LINSTOR上下文中用于标识存储。要对多个节点的存储池进行分组,只需在每个节点上使用相同的名称。例如,一种有效的方法是给所有的ssd取一个名字,给所有的hdd取另一个名字。

在每个提供存储的主机上,您需要创建LVM VG或ZFS zPool。使用一个LINSTOR存储池名称标识的VG和zPool在主机上可能有不同的VG或zPool名称,但请您自己考虑,是否在所有节点上使用相同的VG或zPool名称。

# vgcreate vg_ssd /dev/nvme0n1 /dev/nvme1n1 [...]

然后需要向LINSTOR注册:

# linstor storage-pool create lvm alpha pool_ssd vg_ssd
# linstor storage-pool create lvm bravo pool_ssd vg_ssd
存储池名称和公共元数据称为 存储池定义 。上面列出的命令隐式创建了存储池定义。使用 linstor storage-pool-definition list 可以看到这一点。也可以显式创建存储池定义,但不是必需的。

要列出您可以使用的存储池,请执行以下操作:

# linstor storage-pool list

或者使用短版本

# linstor sp l

如果由于附加的资源或快照(其中一些卷位于另一个仍在运行的存储池中)而阻止删除存储池,则会在相应list命令的 status 列中给出提示(例如 linstor resource list ) 手动删除丢失的存储池中的LINSTOR对象后,可以再次执行lost命令,以确保完全删除了存储池及其剩余对象。

1.11.1. Confining Failure Domains to a Single Back-end Device

In clusters where you have only one kind of storage and the capability to hot swap storage devices, you may choose a model where you create one storage pool per physical backing device. The advantage of this model is to confine failure domains to a single storage device.

1.11.2. Sharing Storage Pools with Multiple Nodes

Both the Exos and LVM2 storage providers offer the option of multiple server nodes directly connected to the storage array and drives. With LVM2 the external locking service (lvmlockd) manages volume groups created with the –shared options with vgcreate. The --shared-space can be used when configuring a LINSTOR pool to use the same LVM2 volume group accessible by two or more nodes. The example below shows using the LVM2 volume group UUID as the shared space identifier for a pool accessible by nodes alpha and bravo:

# linstor storage-pool create lvm --external-locking \
  --shared-space O1btSy-UO1n-lOAo-4umW-ETZM-sxQD-qT4V87 \
  alpha pool_ssd shared_vg_ssd
# linstor storage-pool create lvm --external-locking \
  --shared-space O1btSy-UO1n-lOAo-4umW-ETZM-sxQD-qT4V87 \
  bravo pool_ssd shared_vg_ssd

Exos pools will use the Exos pool serial number by default for the shared-space identifier.

1.11.3. Creating Storage Pools

Since linstor-server 1.5.2 and a recent linstor-client, LINSTOR can create LVM/ZFS pools on a satellite for you. The linstor-client has the following commands to list possible disks and create storage pools, but such LVM/ZFS pools are not managed by LINSTOR and there is no delete command, so such action must be done manually on the nodes.

# linstor physical-storage list

Will give you a list of available disks grouped by size and rotational(SSD/Magnetic Disk).

It will only show disks that pass the following filters:

  • The device size must be greater than 1GiB.

  • The device is a root device (not having children), for example, /dev/vda, /dev/sda.

  • The device does not have any file system or other blkid marker (wipefs -a might be needed).

  • The device is not a DRBD device.

With the create-device-pool command you can create a LVM pool on a disk and also directly add it as a storage-pool in LINSTOR.

# linstor physical-storage create-device-pool --pool-name lv_my_pool \
  LVMTHIN node_alpha /dev/vdc --storage-pool newpool

If the --storage-pool option was provided, LINSTOR will create a storage-pool with the given name.

For more options and exact command usage please check the linstor-client help.

1.12. Using Resource Groups to Deploy LINSTOR Provisioned Volumes

A resource group is a parent object of a resource definition where all property changes made on a resource group will be inherited by its resource definition children. The resource group also stores settings for automatic placement rules and can spawn a resource definition depending on the stored rules.

简单地说,资源组就像模板,定义从它们创建的资源的特性。对这些伪模板的更改将应用于从资源组中创建的所有资源,并具有追溯性。

使用资源组定义资源配置方式应被视为部署由LINSTOR配置的卷的典型方法。后面描述从 资源定义卷定义 创建每个 资源 的章节应仅在特殊情况下使用。
即使您选择不在LINSTOR集群中创建和使用 资源组 ,从 资源定义卷定义 创建的所有资源都将存在于 DfltRscGrp 资源组 中。

使用资源组部署资源的简单模式如下:

# linstor resource-group create my_ssd_group --storage-pool pool_ssd --place-count 2
# linstor volume-group create my_ssd_group
# linstor resource-group spawn-resources my_ssd_group my_ssd_res 20G

上述命令创建出一个名为 my_ssd_res 的资源,其中一个定义了双副本数的20GB卷将从名为 pool_ssd 的存储池的节点自动配置。

一个更有用的模式可能是创建一个资源组,其中的设置是您确定的最适合您的用例的。也许您必须对卷的一致性进行夜间联机验证,在这种情况下,您可以创建一个资源组,其中已经设置了您选择的 verify-alg ,以便从该组生成的资源预先配置为 verify-alg 集:

# linstor resource-group create my_verify_group --storage-pool pool_ssd --place-count 2
# linstor resource-group drbd-options --verify-alg crc32c my_verify_group
# linstor volume-group create my_verify_group
# for i in {00..19}; do
    linstor resource-group spawn-resources my_verify_group res$i 10G
  done

上述命令创建出20个10GiB资源,每个资源都预先配置了 crc32cverify alg

You can tune the settings of individual resources or volumes spawned from resource groups by setting options on the respective resource-definition or volume-definition. For example, if ‘res11’ from the example above is used by a very active database receiving many small random writes, you might want to increase the ‘al-extents’ for that specific resource:

# linstor resource-definition drbd-options --al-extents 6007 res11

If you configure a setting in a resource-definition that is already configured on the resource-group it was spawned from, the value set in the resource-definition will override the value set on the parent resource-group. For example, if the same ‘res11’ was required to use the slower but more secure ‘sha256’ hash algorithm in its verifications, setting the ‘verify-alg’ on the resource-definition for ‘res11’ would override the value set on the resource-group:

# linstor resource-definition drbd-options --verify-alg sha256 res11
A guiding rule for the hierarchy in which settings are inherited is that the value “closer” to the resource or volume wins: volume-definition settings take precedence over volume-group settings, and resource-definition settings take precedence over resource-group settings.

1.13. Configuring a Cluster

1.13.1. Available Storage Plug-ins

LINSTOR has the following supported storage plug-ins at the time of writing:

  • Thick LVM

  • Thin LVM with a single thin pool

  • Thick ZFS

  • Thin ZFS

1.14. Creating and Deploying Resources and Volumes

在下面的场景中,我们假设目标是创建一个大小为 500 GB 的资源 备份,该资源在三个集群节点之间复制。

首先,我们创建一个新的资源定义:

# linstor resource-definition create backups

其次,我们在该资源定义中创建一个新的卷定义:

# linstor volume-definition create backups 500G

如果要更改卷定义的大小,只需执行以下操作:

# linstor volume-definition set-size backups 0 100G

The parameter 0 is the number of the volume in the resource backups. You have to provide this parameter because resources can have multiple volumes that are identified by a so-called volume number. This number can be found by listing the volume-definitions.

The size of a volume-definition can only be decreased if it has no resource. Despite that, the size can be increased even with a deployed resource.

到目前为止,我们只在LINSTOR的数据库中创建了对象,没有在存储节点上创建一个LV。现在你可以选择将安置任务派发给LINSTOR,或者自己做。

1.14.1. Manually Placing Resources

使用 resource create 命令,可以将资源定义显式分配给命名节点。

# linstor resource create alpha backups --storage-pool pool_hdd
# linstor resource create bravo backups --storage-pool pool_hdd
# linstor resource create charlie backups --storage-pool pool_hdd

1.14.2. Automatically Placing Resources

It is possible to have LINSTOR select nodes and storage pools to deploy a resource to, by using the linstor resource create --auto-place or linstor resource-definition auto-place commmands. This section will use the resource create --auto-place command in examples. However, you can use either command to produce the same results.

LINSTOR’s resource-group create command does not have an --auto-place option, because the command does not deploy resources; it only creates a template from which you can later deploy (spawn) resources. However, you can use the arguments described in this section that accompany the --auto-place option with the resource-group create command. When used this way, when you spawn a resource from a resource group, LINSTOR will deploy the resource as if you had used the resource create --auto-place command.
If the nodes (or storage pools) in your cluster cannot fulfill the constraints of your --auto-place command arguments, then LINSTOR will reject your command with an error message.

In the following example, the value after the --auto-place option tells LINSTOR how many replicas you want to have. The storage-pool option should be obvious.

# linstor resource create backups --auto-place 3 --storage-pool pool_hdd

可能不太明显的是,您可以省略 --storage-pool 选项,然后LINSTOR可以自己选择一个存储池。选择遵循以下规则:

  • 忽略当前用户无权访问的所有节点和存储池

  • 忽略所有无磁盘存储池

  • 忽略没有足够可用空间的所有存储池

The remaining storage pools will be rated by different strategies. LINSTOR has currently three strategies:

MaxFreeSpace: This strategy maps the rating 1:1 to the remaining free space of the storage pool. However, this strategy only considers the actually allocated space (in case of thin-provisioned storage pool this might grow with time without creating new resources) MinReservedSpace: Unlike the “MaxFreeSpace”, this strategy considers the reserved space. That is the space that a thin volume can grow to before reaching its limit. The sum of reserved spaces might exceed the storage pool’s capacity, which is as overprovisioning. MinRscCount: Simply the count of resources already deployed in a given storage pool MaxThroughput: For this strategy, the storage pool’s Autoplacer/MaxThroughput property is the base of the score, or 0 if the property is not present. Every Volume deployed in the given storage pool will subtract its defined sys/fs/blkio_throttle_read and sys/fs/blkio_throttle_write property- value from the storage pool’s max throughput. The resulting score might be negative.

The scores of the strategies will be normalized, weighted and summed up, where the scores of minimizing strategies will be converted first to allow an overall maximization of the resulting score.

The weights of the strategies can be configured with the following command:

linstor controller set-property Autoplacer/Weights/$name_of_the_strategy $weight

The strategy names are listed above and the weight can be an arbitrary decimal.

To keep the behavior of the Autoplacer compatible with previous LINSTOR versions, all strategies have a default-weight of 0, except the MaxFreeSpace which has a weight of 1.
Neither 0 nor a negative score will prevent a storage pool from getting selected. A storage pool with these scores will just be considered later.

Finally, LINSTOR tries to find the best matching group of storage pools meeting all requirements. This step also considers other auto-placement restrictions such as --replicas-on-same, --replicas-on-different, --do-not-place-with, --do-not-place-with-regex, --layer-list, and --providers.

Avoiding Colocating Resources When Automatically Placing a Resource

The --do-not-place-with <resource_name_to_avoid> argument specifies that LINSTOR should try to avoid deploying the resource on nodes that already have the specified, resource_name_to_avoid resource deployed.

By using the --do-not-place-with-regex <regular_expression> argument, you can specify that LINSTOR should try to avoid placing the resource on nodes that already have a resource deployed whose name matches the regular expression that you provide with the argument. In this way, you can specify multiple resources to try to avoid placing your resource with.

Constraining Automatic Resource Placement by Using Auxiliary Node Properties

You can constrain automatic resource placement to place (or avoid placing) a resource with nodes having a specified auxiliary node property.

This ability can be particularly useful if you are trying to constrain resource placement within Kubernetes environments that use LINSTOR managed storage. For example, you might set an auxiliary node property that corresponds to a Kubernetes label. See the “replicasOnSame” section within the “LINSTOR Volumes in Kubernetes” LINSTOR User’s Guide chapter for more details about this use case.

The arguments, --replicas-on-same and --replicas-on-different expect the name of a property within the Aux/ namespace.

The following example shows setting an auxiliary node property, testProperty, on three LINSTOR satellite nodes. Next, you create a resource group, testRscGrp, with a place count of two and a constraint to place spawned resources on nodes that have a testProperty value of 1. After creating a volume group, you can spawn a resource from the resource group. For simplicity, output from the following commands is not shown.

# for i in {0,2}; do linstor node set-property --aux node-$i testProperty 1; done
# linstor node set-property --aux node-1 testProperty 0
# linstor resource-group create testRscGrp --place-count 2 --replicas-on-same testProperty=1
# linstor volume-group create testRscGrp
# linstor resource-group spawn-resources testRscGrp testResource 100M

You can verify the placement of the spawned resource by using the following command:

|=====================================================================================|
# linstor resource list
+-------------------------------------------------------------------------------------+
| ResourceName      | Node   | Port | Usage  | Conns |    State | CreatedOn           |
| testResource      | node-0 | 7000 | Unused | Ok    | UpToDate | 2022-07-27 16:14:16 |
| testResource      | node-2 | 7000 | Unused | Ok    | UpToDate | 2022-07-27 16:14:16 |
+-------------------------------------------------------------------------------------+

Because of the --replicas-on-same constraint, LINSTOR did not place the spawned resource on satellite node node-1, because the value of its auxiliary node property, testProperty was 0 and not 1.

You can verify the node properties of node-1, by using the list-properties command:

# linstor node list-properties node-1
+----------------------------+
| Key              | Value   |
|============================|
| Aux/testProperty | 0       |
| CurStltConnName  | default |
| NodeUname        | node-1  |
+----------------------------+
Using Auto-place to Extend Existing Resource Deployments

Besides specifying a positive integer for the --auto-place value for the number of replicas of your resource to place, you can also specify a value of +1, should you want to extend existing resource deployments. By using this value, LINSTOR will create an additional replica, no matter what the --place-count is configured for on the corresponding resource group that the resource was created from.

For example, you can use the +1 auto-place value to deploy an additional replica of the testResource resource used in the previous example. You will first need to set the auxiliary node property, testProperty to 1 on node-1. Otherwise, LINSTOR will not be able to deploy the replica because of the previously configured --replicas-on-same constraint. For simplicity, not all output from the commands below is shown.

# linstor node set-property --aux node-1 testProperty 1
# linstor resource create --auto-place +1 testResource
# linstor resource list
+----+
| ResourceName      | Node   | Port | Usage  | Conns |    State | CreatedOn           |
|=====================================================================================|
| testResource      | node-0 | 7000 | Unused | Ok    | UpToDate | 2022-07-27 16:14:16 |
| testResource      | node-1 | 7000 | Unused | Ok    | UpToDate | 2022-07-28 19:27:30 |
| testResource      | node-2 | 7000 | Unused | Ok    | UpToDate | 2022-07-27 16:14:16 |
+-------------------------------------------------------------------------------------+
The +1 value is not valid for the resource-group create --place-count command. This is because the command does not deploy resources, it only creates templates from which to deploy them later.
Constraining Automatic Resource Placement by LINSTOR Layers or Storage Pool Providers

You can specify the --layer-list or --providers arguments, followed by a comma-separated values (CSV) list of LINSTOR layers or storage pool providers, to influence where LINSTOR places your resource. The possible layers and storage pool providers that you can specify in your CSV list can be shown by using the --help option with the --auto-place option. A CSV list of layers would constrain automatic resource placement for your specified resource to nodes that have storage that conformed with your list. For example, given an existing resource definition named my_luks_resource, consider the following command:

# linstor resource create my_luks_resource --auto-place 3 --layer-list
drbd,luks

This command would create a resource deployed across three nodes having storage pools backed by a DRBD layer backed by a LUKS layer (and implicitly backed by a “storage” layer). The order of layers that you specify in your CSV list is “top-down”, where a layer on the left in the list is above a layer on its right.

The --providers argument can be used to constrain automatic resource placement to only storage pools that match those in a specified CSV list. You can use this argument to have explicit control over which storage pools will back your deployed resource. If for example, you had a mixed environment of ZFS, LVM, and LVM_THIN storage pools in your cluster, by using the --providers LVM,LVM_THIN argument, you can specify that a resource only gets backed by either an LVM or LVM_THIN storage pool, when using the --auto-place option.

The --providers argument’s CSV list does not specify an order of priority for the list elements. Instead, LINSTOR will use factors like additional --auto-place constraints, available free space, and LINSTOR’s storage pool selection strategies that were previously described, when placing a resource.

1.15. Deleting Resources, Resource Definitions, and Resource Groups

You can delete LINSTOR resources, resource definitions, and resource groups by using the delete command after the LINSTOR object that you want to delete. Depending on which object you delete, there will be different implications for your LINSTOR cluster and other associated LINSTOR objects.

1.15.1. Deleting a Resource Definition

You can delete a resource definition by using the command:

# linstor resource-definition delete <resource_definition_name>

This will remove the named resource definition from the entire LINSTOR cluster. The resource is removed from all nodes and the resource entry is marked for removal from LINSTOR’s database tables. After LINSTOR has removed the resource from all the nodes, the resource entry is removed from LINSTOR’s database tables.

If your resource definition has existing snapshots, you will not be able to delete the resource definition until you delete its snapshots. See the Removing a Snapshot section in this guide.

1.15.2. Deleting a Resource

You can delete a resource using the command:

# linstor resource delete <node_name> <resource_name>

Unlike deleting a resource definition, this command will only delete a LINSTOR resource from the node (or nodes) that you specify. The resource is removed from the node and the resource entry is marked for removal from LINSTOR’s database tables. After LINSTOR has removed the resource from the node, the resource entry is removed from LINSTOR’s database tables.

Deleting a LINSTOR resource may have implications for a cluster, beyond just removing the resource. For example, if the resource is backed by a DRBD layer, removing a resource from one node in a three node cluster could also remove certain quorum related DRBD options, if any existed for the resource. After removing such a resource from a node in a three node cluster, the resource would no longer have quorum as it would now only be deployed on two nodes in the three node cluster.

After running a linstor resource delete command to remove a resource from a single node, you might see informational messages such as:

INFO:
    Resource-definition property 'DrbdOptions/Resource/quorum' was removed as there are not enough resources for quorum
INFO:
    Resource-definition property 'DrbdOptions/Resource/on-no-quorum' was removed as there are not enough resources for quorum

Also unlike deleting a resource definition, you can delete a resource while there are existing snapshots of the resource’s storage pool. Any existing snapshots for the resource’s storage pool will persist.

1.15.3. Deleting a Resource Group

You can delete a resource group by using the command:

# linstor resource-group delete <resource_group_name>

As you might expect, this command deletes the named resource group. You can only delete a resource group if it has no associated resource definitions, otherwise LINSTOR will present an error message, such as:

ERROR:
Description:
    Cannot delete resource group 'my_rg' because it has existing resource definitions.

To resolve this error so that you can delete the resource group, you can either delete the associated resource definitions, or your can move the resource definitions to another (existing) resource group:

# linstor resource-definition modify <resource_definition_name> \
--resource-group <another_resource_group_name>

You can find which resource definitions are associated with your resource group by entering the following command:

# linstor resource-definition list

2. Further LINSTOR Tasks

2.1. Creating a Highly Available LINSTOR Cluster

By default a LINSTOR cluster consists of exactly one LINSTOR controller. Making LINSTOR highly available involves providing replicated storage for the controller database, multiple LINSTOR controllers where only one is active at a time, and a service manager that takes care of mounting and unmounting the highly available storage as well as starting and stopping LINSTOR controllers.

2.1.1. Configuring Highly Available Storage

For configuring the highly-available storage we use LINSTOR itself. This has the advantage that the storage is under LINSTOR control and can for example be easily extended to new cluster nodes. Just create a new resource with 200MB in size. This could look like this, you certainly need to adapt the storage pool name:

# linstor resource-definition create linstor_db # linstor
resource-definition drbd-options --on-no-quorum=io-error linstor_db #
linstor resource-definition drbd-options --auto-promote=no linstor_db #
linstor volume-definition create linstor_db 200M # linstor resource create
linstor_db -s pool1 --auto-place 3

From now on we assume the resource’s name is “linstor_db”. It is crucial that your cluster qualifies for auto-quorum and uses the io-error policy (see Section Auto-Quorum Policies), and that auto-promote is disabled.

After the resource is created, it is time to move the LINSTOR DB to the new storage and to create a systemd mount service. First we stop the current controller and disable it, as it will be managed by drbd-reactor later.

# systemctl disable --now linstor-controller

# cat << EOF > /etc/systemd/system/var-lib-linstor.mount
[Unit]
Description=Filesystem for the LINSTOR controller

[Mount]
# you can use the minor like /dev/drbdX or the udev symlink
What=/dev/drbd/by-res/linstor_db/0 Where=/var/lib/linstor EOF

# mv /var/lib/linstor{,.orig} # mkdir /var/lib/linstor # chattr +i
/var/lib/linstor # only if on LINSTOR >= 1.14.0 # drbdadm primary linstor_db
# mkfs.ext4 /dev/drbd/by-res/linstor_db/0 # systemctl start
var-lib-linstor.mount # cp -r /var/lib/linstor.orig/* /var/lib/linstor #
systemctl start linstor-controller

Copy the /etc/systemd/system/var-lib-linstor.mount mount file to all the standby nodes for the linstor controller. Again, do not systemctl enable any of these services, they get managed by drbd-reactor.

2.1.2. Installing Multiple LINSTOR Controllers

The next step is to install LINSTOR controllers on all nodes that have access to the linstor_db DRBD resource (as they need to mount the DRBD volume) and which you want to become a possible LINSTOR controller. It is important that the controllers are manged by drbd-reactor, so verify that the linstor-controller.service is disabled on all nodes! To be sure, execute systemctl disable linstor-controller on all cluster nodes and systemctl stop linstor-controller on all nodes except the one it is currently running from the previous step. Also verify that you have set chattr +i /var/lib/linstor on all potential controller nodes if you use LINSTOR version equal or greater to 1.14.0.

2.1.3. Managing the Services

For starting and stopping the mount service and the linstor-controller service we use drbd-reactor. Install this component on all nodes that could become a LINSTOR controller and edit their /etc/drbd-reactor.d/linstor_db.toml configuration file. It should contain an enabled promoter plug-in section like this:

[[promoter]]
id = "linstor_db"
[promoter.resources.linstor_db]
start = ["var-lib-linstor.mount", "linstor-controller.service"]

Depending on your requirements you might also want to set an on-stop-failure action and set stop-services-on-exit.

After that restart drbd-reactor and enable it on all the nodes you configured it.

# systemctl restart drbd-reactor # systemctl enable drbd-reactor

Check that there are no warnings from drbd-reactor service in the logs by running systemctl status drbd-reactor. As there is already an active LINSTOR controller things will just stay the way they are. Run drbd-reactorctl status linstor_db to check the health of the linstor_db target unit.

The last but nevertheless important step is to configure the LINSTOR satellite services to not delete (and then regenerate) the resource file for the LINSTOR controller DB at its startup. Do not edit the service files directly, but use systemctl edit. Edit the service file on all nodes that could become a LINSTOR controller and that are also LINSTOR satellites.

# systemctl edit linstor-satellite
[Service]
Environment=LS_KEEP_RES=linstor_db

After this change you should execute systemctl restart linstor-satellite on all satellite nodes.

Be sure to configure your LINSTOR client for use with multiple controllers as described in the section titled, Using the LINSTOR Client and verify that you also configured your integration plug-ins (for example, the Proxmox plug-in) to be ready for multiple LINSTOR controllers.

2.2. DRBD Clients

By using the --drbd-diskless option instead of --storage-pool you can have a permanently diskless DRBD device on a node. This means that the resource will appear as block device and can be mounted to the filesystem without an existing storage-device. The data of the resource is accessed over the network on another node with the same resource.

# linstor resource create delta backups --drbd-diskless
The option --diskless was deprecated. Please use --drbd-diskless or --nvme-initiator instead.

2.3. DRBD Consistency Groups (Multiple Volumes within a Resource)

The so called consistency group is a feature from DRBD. It is mentioned in this user’s guide, due to the fact that one of LINSTOR’s main functions is to manage storage-clusters with DRBD. Multiple volumes in one resource are a consistency group.

这意味着一个资源的不同卷上的更改在其他Satellites上以相同的时间顺序复制。

因此,如果资源中的不同卷上有相互依赖的数据,也不必担心时间问题。

要在LINSTOR资源中部署多个卷,必须创建两个同名的卷定义。

# linstor volume-definition create backups 500G # linstor volume-definition
create backups 100G

2.4. Placing Volumes of One Resource in Different Storage Pools

这可以通过在将资源部署到节点之前将 StorPoolName 属性设置为卷定义来实现:

# linstor resource-definition create backups # linstor volume-definition
create backups 500G # linstor volume-definition create backups 100G #
linstor volume-definition set-property backups 0 StorPoolName pool_hdd #
linstor volume-definition set-property backups 1 StorPoolName pool_ssd #
linstor resource create alpha backups # linstor resource create bravo
backups # linstor resource create charlie backups
Since the volume-definition create command is used without the --vlmnr option LINSTOR assigned the volume numbers starting at 0. In the following two lines the 0 and 1 refer to these automatically assigned volume numbers.

Here the ‘resource create’ commands do not need a --storage-pool option. In this case LINSTOR uses a ‘fallback’ storage pool. Finding that storage pool, LINSTOR queries the properties of the following objects in the following order:

  • Volume definition

  • Resource

  • Resource definition

  • Node

If none of those objects contain a StorPoolName property, the controller falls back to a hard-coded ‘DfltStorPool’ string as a storage pool.

This also means that if you forgot to define a storage pool prior deploying a resource, you will get an error message that LINSTOR could not find the storage pool named ‘DfltStorPool’.

2.5. Using LINSTOR Without DRBD

LINSTOR也可以在没有DRBD的情况下使用。没有DRBD,LINSTOR能够从LVM和ZFS支持的存储池中配置卷,并在LINSTOR集群中的各个节点上创建这些卷。

Currently LINSTOR supports the creation of LVM and ZFS volumes with the option of layering some combinations of LUKS, DRBD, or NVMe-oF/NVMe-TCP on top of those volumes.

例如,我们在LINSTOR集群中定义了一个Thin LVM支持的存储池,名为 thin-lvm:

# linstor --no-utf8 storage-pool list
+--------------------------------------------------------------+
| StoragePool | Node      | Driver   | PoolName          | ... |
|--------------------------------------------------------------|
| thin-lvm    | linstor-a | LVM_THIN | drbdpool/thinpool | ... |
| thin-lvm    | linstor-b | LVM_THIN | drbdpool/thinpool | ... |
| thin-lvm    | linstor-c | LVM_THIN | drbdpool/thinpool | ... |
| thin-lvm    | linstor-d | LVM_THIN | drbdpool/thinpool | ... |
+--------------------------------------------------------------+

We could use LINSTOR to create a Thin LVM on linstor-d that’s 100GiB in size using the following commands:

# linstor resource-definition create rsc-1
# linstor volume-definition create rsc-1 100GiB
# linstor resource create --layer-list storage \
          --storage-pool thin-lvm linstor-d rsc-1

You should then see you have a new Thin LVM on linstor-d. You can extract the device path from LINSTOR by listing your linstor resources with the --machine-readable flag set:

# linstor --machine-readable resource list | grep device_path
            "device_path": "/dev/drbdpool/rsc-1_00000",

If you wanted to layer DRBD on top of this volume, which is the default --layer-list option in LINSTOR for ZFS or LVM backed volumes, you would use the following resource creation pattern instead:

# linstor resource-definition create rsc-1
# linstor volume-definition create rsc-1 100GiB
# linstor resource create --layer-list drbd,storage \
          --storage-pool thin-lvm linstor-d rsc-1

You would then see that you have a new Thin LVM backing a DRBD volume on linstor-d:

# linstor --machine-readable resource list | grep -e device_path -e backing_disk
            "device_path": "/dev/drbd1000",
            "backing_disk": "/dev/drbdpool/rsc-1_00000",

The following table shows which layer can be followed by which child-layer:

[cols=”>1,<5″]

Layer

Child layer

DRBD

CACHE, WRITECACHE, NVME, LUKS, STORAGE

CACHE

WRITECACHE, NVME, LUKS, STORAGE

WRITECACHE

CACHE, NVME, LUKS, STORAGE

NVME

CACHE, WRITECACHE, LUKS, STORAGE

LUKS

STORAGE

STORAGE

一个层只能在层列表中出现一次
For information about the prerequisites for the LUKS layer, refer to the Encrypted Volumes section of this User’s Guide.

2.5.1. NVME-OF/NVME-TCP Linstor Layer

NVMe-oF/NVMe-TCP允许LINSTOR将无盘资源连接到具有相同资源的节点,数据存储在NVMe结构上。这就带来了这样一个优势:通过网络访问数据,无需使用本地存储就可以装载资源。在这种情况下,LINSTOR不使用DRBD,因此不会复制LINSTOR提供的NVMe资源,数据存储在一个节点上。

NVMe-oF仅在支持RDMA的网络上工作,NVMe-TCP在每个可以承载IP流量的网络上工作。如果您想了解有关NVMe-oF/NVMe-TCP的NVMe的更多信息,请访问https://www.linbit.com/en/nvme-linstor-swordfish/。

要将NVMe-oF/NVMe-TCP与LINSTOR一起使用,需要在充当Satellite的每个节点上安装包 nvme-cli,并将NVMe-oF/NVMe-TCP用于资源:

If you are not using Ubuntu, use the suitable command for installing packages on your operating system: SLES: zypper; RHEL-family: dnf.
# apt install nvme-cli

要使资源使用NVMe-oF/NVMe-TCP,必须在创建资源定义时提供一个附加参数:

# linstor resource-definition create nvmedata -l nvme,storage
默认情况下,使用DRBD时-l(layer-stack)参数设置为 drbd, storage。如果要创建既不使用NVMe也不使用DRBD的LINSTOR资源,则必须将 -l 参数设置为仅使用 storage

To use NVMe-TCP rather than the default NVMe-oF, the following property needs to be set:

# linstor resource-definition set-property nvmedata NVMe/TRType tcp

The property NVMe/TRType can alternatively be set on resource-group or controller level.

Next, create the volume-definition for our resource:

# linstor volume-definition create nvmedata 500G

在节点上创建资源之前,必须知道数据将在本地存储在哪里,以及哪个节点通过网络访问它。

首先,我们在存储数据的节点上创建资源:

# linstor resource create alpha nvmedata --storage-pool pool_ssd

在将通过网络访问资源数据的节点上,必须将资源定义为无盘:

# linstor resource create beta nvmedata --nvme-initiator

现在,您可以在一个节点上装载资源 nvmedata

If your nodes have more than one NIC you should force the route between them for NVMe-of/NVME-TCP, otherwise multiple NICs could cause troubles.

2.5.2. OpenFlex™ Layer

Since version 1.5.0 the additional Layer openflex can be used in LINSTOR. From LINSTOR’s perspective, the OpenFlex Composable Infrastructure takes the role of a combined layer acting as a storage layer (like LVM) and also providing the allocated space as an NVMe target. OpenFlex has a REST API which is also used by LINSTOR to operate with.

As OpenFlex combines concepts of LINSTOR’s storage as well as NVMe-layer, LINSTOR was added both, a new storage driver for the storage pools as well as a dedicated openflex layer which uses the mentioned REST API.

In order for LINSTOR to communicate with the OpenFlex-API, LINSTOR needs some additional properties, which can be set once on controller level to take LINSTOR-cluster wide effect:

  • StorDriver/Openflex/ApiHost specifies the host or IP of the API entry-point

  • StorDriver/Openflex/ApiPort this property is glued with a colon to the previous to form the basic http://ip:port part used by the REST calls StorDriver/Openflex/UserName the REST username

  • StorDriver/Openflex/UserPassword the password for the REST user

Once that is configured, we can now create LINSTOR objects to represent the OpenFlex architecture. The theoretical mapping of LINSTOR objects to OpenFlex objects are as follows: Obviously an OpenFlex storage pool is represented by a LINSTOR storage pool. As the next thing above a LINSTOR storage pool is already the node, a LINSTOR node represents an OpenFlex storage device. The OpenFlex objects above storage device are not mapped by LINSTOR.

When using NVMe, LINSTOR was designed to run on both sides, the NVMe target as well as on the NVMe initiator side. In the case of OpenFlex, LINSTOR cannot (or even should not) run on the NVMe target side as that is completely managed by OpenFlex. As LINSTOR still needs nodes and storage pools to represent the OpenFlex counterparts, the LINSTOR client was extended with special node create commands since 1.0.14. These commands not only accept additionally needed configuration data, but also starts a “special satellite” besides the already running controller instance. These special satellites are completely LINSTOR managed. They will shut down when the controller shuts down and will be started again when the controller starts.

The new client command for creating a “special satellite” representing an OpenFlex storage device is:

$ linstor node create-openflex-target ofNode1 192.168.166.7 000af795789d

The arguments are as follows:

  • ofNode1 is the node name which is also used by the standard linstor node create command

  • 192.168.166.7 is the address on which the provided NVMe devices can be accessed. As the NVMe devices are accessed by a dedicated network interface, this address differs from the address specified with the property StorDriver/Openflex/ApiHost. The latter is used for the management / REST API.

  • 000af795789d is the identifier for the OpenFlex storage device.

The last step of the configuration is the creation of LINSTOR storage pools:

$ linstor storage-pool create openflex ofNode1 sp0 0
  • ofNode1 and sp0 are the node name and storage pool name, respectively, just as usual for the LINSTOR’s create storage pool command

  • The last 0 is the identifier of the OpenFlex storage pool within the previously defined storage device

Once all necessary storage pools are created in LINSTOR, the next steps are similar to the usage of using an NVMe resource with LINSTOR. Here is a complete example:

# set the properties once
linstor controller set-property StorDriver/Openflex/ApiHost 10.43.7.185
linstor controller set-property StorDriver/Openflex/ApiPort 80
linstor controller set-property StorDriver/Openflex/UserName myusername
linstor controller set-property StorDriver/Openflex/UserPassword mypassword

# create a node for openflex storage device "000af795789d"
linstor node create-openflex-target ofNode1 192.168.166.7 000af795789d

# create a usual linstor satellite. later used as nvme initiator
linstor node create bravo

# create a storage pool for openflex storage pool "0" within storage device "000af795789d"
linstor storage-pool create openflex ofNode1 sp0 0

# create resource- and volume-definition
linstor resource-definition create backupRsc
linstor volume-definition create backupRsc 10G

# create openflex-based nvme target
linstor resource create ofNode1 backupRsc --storage-pool sp0 --layer-list openflex

# create openflex-based nvme initiator
linstor resource create bravo backupRsc --nvme-initiator --layer-list openflex
In case a node should access the OpenFlex REST API through a different host than specified with + linstor controller set-property StorDriver/Openflex/ApiHost 10.43.7.185 you can always use LINSTOR’s inheritance mechanism for properties. That means simply define the same property on the node-level you need it, i.e. + linstor node set-property ofNode1 StorDriver/Openflex/ApiHost 10.43.8.185

2.5.3. 写入缓存层

A DM-Writecache device is composed of two devices: one storage device and one cache device. LINSTOR can setup such a writecache device, but needs some additional information, like the storage pool and the size of the cache device.

# linstor storage-pool create lvm node1 lvmpool drbdpool
# linstor storage-pool create lvm node1 pmempool pmempool

# linstor resource-definition create r1
# linstor volume-definition create r1 100G

# linstor volume-definition set-property r1 0 Writecache/PoolName pmempool
# linstor volume-definition set-property r1 0 Writecache/Size 1%

# linstor resource create node1 r1 --storage-pool lvmpool --layer-list WRITECACHE,STORAGE

The two properties set in the examples are mandatory, but can also be set on controller level which would act as a default for all resources with WRITECACHE in their --layer-list. However, please note that the Writecache/PoolName refers to the corresponding node. If the node does not have a storage-pool named pmempool you will get an error message.

The 4 mandatory parameters required by DM-Writecache are either configured through a property or figured out by LINSTOR. The optional properties listed in the mentioned link can also be set through a property. Please see linstor controller set-property --help for a list of Writecache/* property-keys.

使用 --layer-list DRBD,WRITECACHE,STORAGE 将DRBD配置为使用外部元数据时,只有备份设备将使用writecache,而不是保存外部元数据的设备。

2.5.4. Cache Layer

LINSTOR can also setup a DM-Cache device, which is very similar to the DM-Writecache from the previous section. The major difference is that a cache device is composed by three devices: one storage device, one cache device and one meta device. The LINSTOR properties are quite similar to those of the writecache but are located in the Cache namespace:

# linstor storage-pool create lvm node1 lvmpool drbdpool
# linstor storage-pool create lvm node1 pmempool pmempool

# linstor resource-definition create r1
# linstor volume-definition create r1 100G

# linstor volume-definition set-property r1 0 Cache/CachePool pmempool
# linstor volume-definition set-property r1 0 Cache/Cachesize 1%

# linstor resource create node1 r1 --storage-pool lvmpool --layer-list CACHE,STORAGE
Rather than Writecache/PoolName (as when configuring the Writecache layer) the Cache layer’s only required property is called Cache/CachePool. The reason for this is that the Cache layer also has a Cache/MetaPool which can be configured separately or it defaults to the value of Cache/CachePool.

Please see linstor controller set-property --help for a list of Cache/* property-keys and default values for omitted properties.

Using --layer-list DRBD,CACHE,STORAGE while having DRBD configured to use external metadata, only the backing device will use a cache, not the device holding the external metadata.

2.5.5. Storage Layer

The storage layer will provide new devices from well known volume managers like LVM, ZFS or others. Every layer combination needs to be based on a storage layer, even if the resource should be diskless – for that type there is a dedicated diskless provider type.

For a list of providers with their properties please see Storage Providers.

For some storage providers LINSTOR has special properties:

  • StorDriver/WaitTimeoutAfterCreate: If LINSTOR expects a device to appear after creation (for example after calls of lvcreate, zfs create,…​), LINSTOR waits per default 500ms for the device to appear. These 500ms can be overridden by this property.

  • StorDriver/dm_stats: If set to true LINSTOR calls dmstats create $device after creation and dmstats delete $device --allregions after deletion of a volume. Currently only enabled for LVM and LVM_THIN storage providers.

2.6. Storage Providers

LINSTOR has a few storage providers. The most used ones are LVM and ZFS. But also for those two providers there are already sub-types for their thin-provisioned variants.

  • Diskless: This provider type is mostly required to have a storage pool that can be configured with LINSTOR properties like PrefNic as described in Managing Network Interface Cards.

  • LVM / LVM-Thin: The adminstrator is expected to specify the LVM volume group or the thin-pool (in form of “LV/thinpool”) to use the corresponding storage type. These drivers support following properties for fine-tuning:

    • StorDriver/LvcreateOptions: The value of this property is appended to every lvcreate …​ call LINSTOR executes.

  • ZFS / ZFS-Thin: The administrator is expected to specify the ZPool that LINSTOR should use. These drivers support following properties for fine-tuning:

    • StorDriver/ZfscreateOptions: The value of this property is appended to every zfs create …​ call LINSTOR executes.

  • File / FileThin: Mostly used for demonstration / experiments. LINSTOR will basically reserve a file in a given directory and will configure a loop device on top of that file.

  • OpenFlex: This special storage provider currently requires to be run on a “special satellite”. Please see OpenFlex™ Layer for more details.

  • EXOS: This special storage provider is currently required to be run on a “special satellite”. Please see the EXOS Integration chapter.

  • SPDK: The administrator is expected to speicify the logical volume store which LINSTOR should use. The usage of this storage provider implies the usage of the NVME Layer.

    • Remote-SPDK: This special storage provider currently requires to be run on a “special satellite”. Please see Remote SPDK Provider for more details.

2.6.1. Remote SPDK Provider

A storage pool with the type remote SPDK can only be created on a “special satellite”. For this you first need to start a new satellite using the command:

$ linstor node create-remote-spdk-target nodeName 192.168.1.110

This will start a new satellite instance running on the same machine as the controller. This special satellite will do all the REST based RPC communication towards the remote SPDK proxy. As the help message of the LINSTOR command shows, the administrator might want to use additional settings when creating this special satellite:

$ linstor node create-remote-spdk-target -h
usage: linstor node create-remote-spdk-target [-h] [--api-port API_PORT]
                                              [--api-user API_USER]
                                              [--api-user-env API_USER_ENV]
                                              [--api-pw [API_PW]]
                                              [--api-pw-env API_PW_ENV]
                                              node_name api_host

The difference between the --api-* and their corresponding --api-\*-env versions is that the version with the -env ending will look for an environment variable containing the actual value to use whereas the --api-\* version directly take the value which is stored in the LINSTOR property. Administrators might not want to save the --api-pw in plain text, which would be clearly visible using commands like linstor node list-property <nodeName>.

Once that special satellite is up and running the actual storage pool can be created:

$ linstor storage-pool create remotespdk -h
usage: linstor storage-pool create remotespdk [-h]
                                              [--shared-space SHARED_SPACE]
                                              [--external-locking]
                                              node_name name driver_pool_name

Whereas node_name is self-explanatory, name is the name of the LINSTOR storage pool and driver_pool_name refers to the SPDK logical volume store.

Once this remotespdk storage pool is created the remaining procedure is quite similar as using NVMe: First the target has to be created by creating a simple “diskful” resource followed by a second resource having the --nvme-initiator option enabled.

2.7. 管理网络接口卡

LINSTOR can deal with multiple network interface cards (NICs) in a machine. They are called “net interfaces” in LINSTOR speak.

When a satellite node is created a first net interface gets created implicitly with the name default. You can use the --interface-name option of the node create command to give it a different name, when you create the satellite node.

For existing nodes, additional net interfaces are created like this:

# linstor node interface create node-0 10G_nic 192.168.43.231

Net interfaces are identified by the IP address only, the name is arbitrary and is not related to the NIC name used by Linux. You can then assign the net interface to a node so that the node’s DRBD traffic will be routed through the corresponding NIC.

# linstor node set-property node-0 PrefNic 10G_nic
It is also possible to set the PrefNic property on a storage pool. DRBD traffic from resources using the storage pool will be routed through the corresponding NIC. However, you need to be careful here. Any DRBD resource that requires Diskless storage, for example, diskless storage acting in a tiebreaker role for DRBD quorum purposes, will go through the default satellite node net interface, until you also set the PrefNic property for the default net interface. Setups can become complex. It is far easier and safer, if you can get away with it, to set the PrefNic property at the node level. This way, all storage pools on the node, including Diskless storage pools, will use your preferred NIC.

If you need to add an interface for only controller-satellite traffic, you can add an interface using the above node interface create command. Then you modify the connection to make it the active controller-satellite connection. For example, if you added an interface named 1G-satconn on all nodes, after adding the interface, you can then tell LINSTOR to use this interface for controller-satellite traffic by entering the following command:

# linstor node interface modify node-0 1G-satconn --active

You can verify this change by using the linstor node interface list node-0 command. Output from the command should show that the StltCon label applies to the 1G-satconn interface.

While the above methods can re-route DRBD traffic, and LINSTOR communication between the controller and satellite nodes, it is not possible through linstor commands only, to route LINSTOR controller-client traffic (commands that you issue from a LINSTOR client to the controller) through a specific NIC. To achieve this, you can either:

  1. Specify a LINSTOR controller using methods outlined in Using the LINSTOR Client and have the only route to the controller as specified be through the NIC that you want to use for controller-client traffic.

  2. Use Linux tools such as ip route and iptables to filter LINSTOR client-controller traffic, port number 3370, and route it through a specific NIC.

2.7.1. Creating Multiple DRBD Paths with LINSTOR

To use multiple network paths for DRBD setups, the PrefNic property is not sufficient. Instead the linstor node interface and linstor resource-connection path commands should be used, as shown below.

# linstor node interface create alpha nic1 192.168.43.221
# linstor node interface create alpha nic2 192.168.44.221
# linstor node interface create bravo nic1 192.168.43.222
# linstor node interface create bravo nic2 192.168.44.222

# linstor resource-connection path create alpha bravo myResource path1 nic1 nic1
# linstor resource-connection path create alpha bravo myResource path2 nic2 nic2

In the example above we define two network interfaces (nic1 and nic2) for each node. The last two commands create network path entries in the generated DRBD .res file. This is the relevant part of the resulting .res file:

resource myResource {
  ...
  connection {
    path {
      host alpha address 192.168.43.221:7000;
      host bravo address 192.168.43.222:7000;
    }
    path {
      host alpha address 192.168.44.221:7000;
      host bravo address 192.168.44.222:7000;
    }
  }
}
While it is possible to specify a port number to be used for LINSTOR satellite traffic when creating a node interface, this port number is ignored when creating a DRBD resource connection path. Instead, the command will assign a port number dynamically, starting from port number 7000 and incrementing up.

2.8. Encrypted Volumes

LINSTOR can handle transparent encryption of DRBD volumes. dm-crypt is used to encrypt the provided storage from the storage device.

To use dm-crypt please verify that cryptsetup is installed before you start the satellite.

使用加密的基本步骤:

  1. 创建主密码

  2. Add luks to the layer-list. Note that all plug-ins (e.g., Proxmox) require a DRBD layer as the top most layer if they do not explicitly state otherwise.

  3. 不要忘记在controller重新启动后重新输入主密码。

2.8.1. Encryption Commands

下面是有关命令的详细信息。

在LINSTOR可以加密任何卷之前,需要创建主密码。这可以通过linstor客户端完成。

# linstor encryption create-passphrase

`crypt create-passphrase`将等待用户输入初始主密码(因为所有其他crypt命令都没有参数)。

如果您想更改主密码,可以使用以下方法:

# linstor encryption modify-passphrase

luks 层可以在创建资源定义或资源本身时添加,而建议使用前一种方法,因为它将自动应用于从该资源定义创建的所有资源。

# linstor resource-definition create crypt_rsc --layer-list luks,storage

要输入主密码(在controller重新启动后),请使用以下命令:

# linstor encryption enter-passphrase
无论何时重新启动linstor controller,用户都必须向控制器发送主密码,否则linstor无法重新打开或创建加密卷。

2.8.2. Automatic Passphrase

It is possible to automate the process of creating and re-entering the master passphrase.

To use this, either an environment variable called MASTER_PASSPHRASE or an entry in /etc/linstor/linstor.toml containing the master passphrase has to be created.

The required linstor.toml looks like this:

[encrypt]
passphrase="example"

If either one of these is set, then every time the controller starts it will check whether a master passphrase already exists. If there is none, it will create a new master passphrase as specified. Otherwise, the controller enters the passphrase.

If a master passphrase is already configured, and it is not the same one as specified in the environment variable or linstor.toml, the controller will be unable to re-enter the master passphrase and react as if the user had entered a wrong passphrase. This can only be resolved through manual input from the user, using the same commands as if the controller was started without the automatic passphrase.
In case the master passphrase is set in both an environment variable and the linstor.toml, only the master passphrase from the linstor.toml will be used.

2.9. Checking Cluster State

LINSTOR provides various commands to check the state of your cluster. These commands start with a ‘list’ precursor, after which, various filtering and sorting options can be used. The ‘–groupby’ option can be used to group and sort the output in multiple dimensions.

# linstor node list
# linstor storage-pool list --groupby Size

2.10. Evacuating a Node

You can use the LINSTOR command node evacuate to evacuate a node of its resources, for example, if you are preparing to delete a node from your cluster, and you need the node’s resources moved to other nodes in the cluster. After successfully evacuating a node, the node’s LINSTOR status will show as “EVACUATE” rather than “Online”, and it will have no LINSTOR resources on it.

If you are evacuating a node where LINSTOR is deployed within another environment, such as Kubernetes, or OpenNebula, you need to move the node’s LINSTOR-backed workload to another node in your cluster before evacuating its resources. For special actions and considerations within a Kubernetes environment, see the Evacuating a Node in Kubernetes section. For a LINSTOR node in OpenNebula, you need to perform a live migration of the OpenNebula LINSTOR-backed virtual machines that your node hosts, to another node in your cluster, before evacuating the node’s resources.

Evacuate a node using the following steps:

  1. Determine if any resources on the node that you want to evacuate are “InUse”. The “InUse” status corresponds to a resource being in a DRBD Primary state. Before you can evacuate a node successfully, none of the resources on the node should be “InUse”, otherwise LINSTOR will fail to remove the “InUse” resources from the node as part of the evacuation process.

  2. Run linstor node evacuate <node_name>. You will get a warning if there is no suitable replacement node for a resource on the evacuating node. For example, if you have three nodes and you want to evacuate one, but your resource group sets a placement count of three, you will get a warning that will prevent the node from removing the resources from the evacuating node.

  3. Verify that the status of linstor node list for your node is “EVACUATE” rather than “Online”.

  4. Check the “State” status of resources on your node, by using the linstor resource list command. You should see syncing activity that will last for sometime, depending on the size of the data sets in your node’s resources.

  5. List the remaining resources on the node by using the command linstor resource list --nodes <node_name>. If any are left, verify whether they are just waiting for the sync to complete.

  6. Verify that there are no resources on the node, by using the linstor resource list command.

  7. Remove the node from the cluster by using the command linstor node delete <node_name>.

2.10.1. Evacuating Multiple Nodes

Some evacuation cases may need special planning. For example, if you are evacuating more than one node, you can exclude the nodes from participating in LINSTOR’s resource autoplacer. You can do this by using the following command on each node that you want to evacuate:

# linstor node set-property <node_name> AutoplaceTarget false

This ensures that LINSTOR will not place resources from a node that you are evacuating onto another node that you plan on evacuating.

2.10.2. Restoring an Evacuating Node

If you already ran a node evacuate command that has either completed or still has resources in an “Evacuating” state, you can remove the “Evacuating” state from a node by using the node restore command. This will work so long as you have not yet run a node delete command.

After restoring the node, you should use the node set-property <node_name> AutoplaceTarget true command, if you previously set the AutoplaceTarget property to “false”. This way, LINSTOR can again place resources onto the node automatically, to fulfill placement count properties that you might have set for resources in your cluster.

If LINSTOR has already evacuated resources when running a node restore command, evacuated resources will not automatically return to the node. If LINSTOR is still in the process of evacuating resources, this process will continue until LINSTOR has placed the resources on other nodes. You will need to manually “move” the resources that were formerly on the restored node. You can do this by first creating the resources on the restored node and then deleting the resources from another node where LINSTOR may have placed them. You can use the resource list command to show you on which nodes your resources are placed.

2.11. Managing Snapshots

精简LVM和ZFS存储池支持快照。

2.11.1. Creating a Snapshot

假设在某些节点上放置了名为 resource1 的资源定义,则可以按如下方式创建快照:

# linstor snapshot create resource1 snap1

这将在资源所在的所有节点上创建快照。LINSTOR将确保即使在资源处于活动使用状态时也创建一致的快照。

Setting the resource-definition property AutoSnapshot/RunEvery LINSTOR will automatically create snapshots every X minute. The optional property AutoSnapshot/Keep can be used to clean-up old snapshots which were created automatically. No manually created snapshot will be cleaned-up / deleted. If AutoSnapshot/Keep is omitted (or ⇐ 0), LINSTOR will keep the last 10 snapshots by default.

# linstor resource-definition set-property AutoSnapshot/RunEvery 15
# linstor resource-definition set-property AutoSnapshot/Keep 5

2.11.2. Restoring a Snapshot

以下步骤将快照还原到新资源。即使原始资源已从创建快照的节点中删除,也可能发生这种情况。

首先使用与快照中的卷匹配的卷定义新资源:

# linstor resource-definition create resource2
# linstor snapshot volume-definition restore --from-resource resource1 \
  --from-snapshot snap1 --to-resource resource2

此时,如果需要,可以应用其他配置。然后,准备好后,根据快照创建资源:

# linstor snapshot resource restore --from-resource resource1 \
  --from-snapshot snap1 --to-resource resource2

这将在快照所在的所有节点上放置新资源。也可以显式选择放置资源的节点;请参阅帮助( linstor snapshot resource restore -h ).

2.11.3. Rolling Back to a Snapshot

LINSTOR可以将资源回滚到快照状态。回滚时资源不能被使用。也就是说,它不能安装在任何节点上。如果资源正在使用中,请考虑是否可以通过restoring the snapshot来实现您的目标。

回滚执行如下:

# linstor snapshot rollback resource1 snap1

资源只能回滚到最近的快照。要回滚到旧快照,请首先删除中间快照。

2.11.4. Removing a Snapshot

可以按如下方式删除现有快照:

# linstor snapshot delete resource1 snap1

2.11.5. Shipping a Snapshot

Snapshots can be shipped between LINSTOR nodes or between different LINSTOR clusters, as well as to an S3 storage such as Amazon S3 or min.io.

The following tools need to be installed on the satellites that are going to send or receive snapshots:

  • zstd is needed to compress the data before it is being shipped

  • thin-send-recv is needed to ship data when using lvm-thin

The satellite needs to be restarted after installing these tools, otherwise LINSTOR will not be able to use them.
Remotes

In a LINSTOR cluster, the definition of a shipping target is called a remote. Currently, there are two different types of remotes: LINSTOR remotes and S3 remotes. LINSTOR remotes are used to ship snapshots to a different LINSTOR cluster, while S3 remotes are needed to ship snapshots to AWS S3, min.io or any other service using S3 compatible object storage.

Since a remote needs to store sensitive data, such as passwords, it is neccessary to have encryption enabled whenever you want to use a remote in any way. How to set up LINSTOR’s encryption is described here.

To create an S3 remote, LINSTOR will need to know the endpoint (that is, the URL of the target S3 server), the name of the target bucket, the region the S3 server is in, as well as the access-key and secret-key used to access the bucket. If the command is sent without adding the secret-key, a prompt will pop up to enter it in. The command should look like this:

# linstor remote create s3 myRemote s3.us-west-2.amazonaws.com \
  my-bucket us-west-2 admin password
Usually, LINSTOR uses the endpoint and bucket to create an URL using the virtual-hosted-style for its access to the given bucket (for example my-bucket.s3.us-west-2.amazonaws.com). Should your setup not allow access this way, change the remote to path-style access (for example s3.us-west-2.amazonaws.com/my-bucket) by adding the --use-path-style argument to make LINSTOR combine the parameters accordingly.

To create a LINSTOR remote, only the URL or IP address of the controller of the target cluster is needed. The command goes as follows:

# linstor remote create linstor myRemote 192.168.0.15

Additionally, to ship LUKS-based (encrypted) backups, it is necessary to add the --passphrase and --cluster-id arguments to the command. This is used to save the passphrase and cluster ID of the target cluster to the remote respectively. For more details on shipping LUKS-based backups between two LINSTOR clusters, see this chapter.

To see all the remotes known to the local cluster, use linstor remote list. To delete a remote, use linstor remote delete myRemoteName. Should an existing remote need altering, use linstor remote modify to change it.

Shipping Snapshots to S3

All that is needed to ship a snapshot to S3 is to create an S3-remote that the current cluster can reach as well as the resource that should be shipped. Then, simply use the following command to ship it there:

# linstor backup create myRemote myRsc

This command will create a snapshot of your resource and ship it to the given remote. If this isn’t the first time you shipped a backup of this resource (to that remote) and the snapshot of the previous backup hasn’t been deleted yet, an incremental backup will be shipped. To force the creation of a full backup, add the --full argument to the command. Getting a specific node to ship the backup is also possible by using --node myNode, but if the specified node is not available or only has the resource diskless, a different node will be chosen.

To see which backups exist in a specific remote, use linstor backup list myRemote. A resource-name can be added to the command as a filter to only show backups of that specific resource by using the argument --resource myRsc. If you use the --other argument, only entries in the bucket that LINSTOR does not recognize as a backup will be shown. LINSTOR always names backups in a certain way, and as long as an item in the remote is named according to this schema, it is assumed that it is a backup created by LINSTOR – so this list will show everything else.

There are several options when it comes to deleting backups:

  • linstor backup delete all myRemote: This command deletes ALL S3-objects on the given remote, provided that they are recognized to be backups, that is, fit the expected naming schema. There is the option --cluster to only delete backups that were created by the current cluster.

  • linstor backup delete id myRemote my-rsc_back_20210824_072543: This command deletes a single backup from the given remote – namely the one with the given id, which consists of the resource-name, the automatically generated snapshot-name (back_timestamp) and, if set, the backup-suffix. The option --prefix lets you delete all backups starting with the given id. The option --cascade deletes not only the specified backup, but all other incremental backups depending on it.

  • linstor backup delete filter myRemote …​: This command has a few different arguments to specify a selection of backups to delete. -t 20210914_120000 will delete all backups made before 12 o’clock on the 14th of September, 2021. -n myNode will delete all backups uploaded by the given node. -r myRsc will delete all backups with the given resource name. These filters can be combined as needed. Finally, --cascade deletes not only the selected backup(s), but all other incremental backups depending on any of the selected backups.

  • linstor backup delete s3key myRemote randomPictureInWrongBucket: This command will find the object with the given S3-key and delete it – without considering anything else. This should only be used to either delete non-backup items from the remote, or to clean up a broken backup that is no longer deleteable by other means. Using this command to delete a regular, working backup will break that backup, so beware!

All commands that have the --cascade option will NOT delete a backup that has incremental backups depending on it unless you explicitly add that option.
All linstor backup delete …​ commands have the --dry-run option, which will give you a list of all the S3-objects that will be deleted. This can be used to ensure nothing that should not be deleted is accidentally deleted.

Maybe the most important task after creating a backup is restoring it. To do so, only the remote is needed – but it is also possible to restore into an existing resource definition with no existing snapshots nor resources. There are two options for the command:

# linstor backup restore myRemote myNode targetRsc --resource sourceRsc
# linstor backup restore myRemote myNode targetRsc --id sourceRsc_back_20210824_072543

Either --resource (-r) or --id must be used, but you cannot use both of them together. -r is used to restore the latest backup of the resource specified with this option, while --id restores the exact backup specified by the given id, and can therefore be used to restore backups other than the most recent.

If the backup to be restored includes a LUKS layer, the --passphrase argument is required. With it, the passphrase of the original cluster of the backup needs to be set so that LINSTOR can decrypt the volumes after download and re-encrypt them with the local passphrase.

The backup restore will download all the snapshots from the last full backup up to the specified backup. Afterwards, it restores the snapshots into a new resource. If that last step should be skipped, the --download-only option needs to be added to the command.

Backups can be downloaded from any cluster, not just the one that uploaded them, provided that the setup is correct. Specifically, the target resource cannot have any existing resources or snapshots, and the storage pool(s) used need to have the same storage providers. If the storage pool(s) on the target node have the exact same names as on the cluster the backup was created on, no extra action is necessary. Should they have different names, the option --storpool-rename needs to be used. It expects at least one oldname=newname pair. For every storage pool of the original backup that is not named in that list, it will be assumed that its name is exactly the same on the target node.

To find out exactly which storage pools need to be renamed, as well as how big the download and the restored resource will be, the command linstor backup info myRemote …​ can be used. Similar to the restore command, either -r or --id need to be given, which add the same restrictions as with that command. To see how much space will be left over in the local storage pools after a restore, the argument -n myNode needs to be added. Just like with a restore, it assumes the storage pool names are exactly the same on the given node as with the backup. Should that not be the case, again, just like with the restore command, --storpool-rename should be used.

Shipping Snapshots Between Two LINSTOR Clusters

Shipping a snapshot directly between two LINSTOR clusters can be done with a LINSTOR remote as well as a resource definition with at least one diskful resource on the source side (where the shipping command is issued). On the target side, you need to create a LINSTOR remote with the cluster ID of the source (remote) cluster:

$ linstor remote create linstor --cluster-id <SOURCE_CLUSTER_ID> <NAME> <URL>
If you do not specify the cluster ID of your source cluster when you create a LINSTOR remote on your target cluster, you will receive an “Unknown Cluster” error when you try to ship a backup. To get the cluster ID of your source cluster, you can enter the command linstor controller list-properties|grep -i cluster from the source cluster.

In the remote create command shown above, <NAME> is an arbitrary name that you specify to identify the remote. <URL> is either the IP address of the source (remote) LINSTOR controller or its resolvable hostname. If you have configured a highly available LINSTOR controller, use its virtual IP address (VIP) or the VIP’s resolvable name.

Snapshot Shipping Within a Single LINSTOR Cluster

If you want to ship a snapshot inside the same cluster, you just need to create a LINSTOR remote that points to the local controller.

Specifying a LINSTOR Passphrase When Creating a Remote

When the snapshot that you want to ship contains a LUKS layer, the remote on the target cluster also needs the passphrase of the source cluster set when you create the remote. This is because the LINSTOR passphrase is used to encrypt the LUKS passphrase. To specify the source cluster’s LINSTOR passphrase when you create a LINSTOR remote on the target cluster, enter:

$ linstor --controllers <TARGET_CONTROLLER> remote create linstor \
--cluster-id <SOURCE_CLUSTER_ID> --passphrase <SOURCE_CONTROLLER_PASSPHRASE> <NAME> <URL>

For LINSTOR to LINSTOR snapshot shipping, you must also create a LINSTOR remote on the source cluster. For simplicity sake, although not strictly necessary, you can specify the target cluster’s LINSTOR passphrase when you create a LINSTOR remote for the target cluster on the source cluster, before you ship backups or snapshots. On the source cluster, enter:

$ linstor --controllers <SOURCE_CONTROLLER> remote create linstor \
--cluster-id <TARGET_CLUSTER_ID> --passphrase <TARGET_CONTROLLER_PASSPHRASE> <NAME> <URL>
If you are specifying a LINSTOR controller node (perhaps because you have a highly available controller), when creating a remote, you can specify the controller either by an IP address or a resolvable hostname.
Shipping a Backup of a LINSTOR Resource

The command to ship a backup is:

# linstor backup ship myRemote localRsc targetRsc

Additionally, you can use --source-node and --target-node to specify which node should send and receive the backup respectively. In case those nodes are not available, a different one will be chosen automatically.

If targetRsc is already a deployed resource on the remote cluster, snapshots in the backup shipping for localRsc will ship to the remote cluster but they will not be restored to the remote cluster. The same is true if you specify the --download-only option with the linstor backup ship command.

Shipping a Snapshot in the Same Cluster

Both, the source as well as the target node have to have the resource for snapshot shipping deployed. Additionally, the target resource has to be deactivated.

# linstor resource deactivate nodeTarget resource1
Deactivating a resource with DRBD in its layer-list can NOT be reactivated again. However, a successfully shipped snapshot of a DRBD resource can still be restored into a new resource.

To manually start the snapshot-shipping, use:

# linstor snapshot ship --from-node nodeSource --to-node nodeTarget --resource resource1
The snapshot ship command is considered deprecated and any bugs found with it will not be fixed. Instead, use the backup ship command with a remote pointing to your local controller. For more details, see the previous section.

By default, the snapshot-shipping uses TCP ports from the range 12000-12999. To change this range, the property SnapshotShipping/TcpPortRange, which accepts a to-from range, can be set on the controller:

# linstor controller set-property SnapshotShipping/TcpPortRange 10000-12000

A resource can also be periodically shipped. To accomplish this, it is mandatory to set the properties SnapshotShipping/TargetNode as well as SnapshotShipping/RunEvery on the resource-definition. SnapshotShipping/SourceNode can also be set, but if omitted LINSTOR will choose an active resource of the same resource-definition.

To allow incremental snapshot-shipping, LINSTOR has to keep at least the last shipped snapshot on the target node. The property SnapshotShipping/Keep can be used to specify how many snapshots LINSTOR should keep. If the property is not set (or ⇐ 0) LINSTOR will keep the last 10 shipped snapshots by default.

# linstor resource-definition set-property resource1 SnapshotShipping/TargetNode nodeTarget
# linstor resource-definition set-property resource1 SnapshotShipping/SourceNode nodeSource
# linstor resource-definition set-property resource1 SnapshotShipping/RunEvery 15
# linstor resource-definition set-property resource1 SnapshotShipping/Keep 5

2.12. Scheduled Backup Shipping

Starting with LINSTOR Controller version 1.19.0 and working with LINSTOR client version 1.14.0 or above, you can configure scheduled backup shipping for deployed LINSTOR resources.

Scheduled backup shipping consists of three parts:

  • A data set that consists of one or more deployed LINSTOR resources that you want to backup and ship

  • A remote destination to ship backups to (another LINSTOR cluster or an S3 instance)

  • A schedule that defines when the backups should ship

LINSTOR backup shipping only works for deployed LINSTOR resources that are backed by LVM and ZFS storage pools, because these are the storage pool types with snapshot support in LINSTOR.

2.12.1. Creating a Backup Shipping Schedule

You create a backup shipping schedule by using the LINSTOR client schedule create command and defining the frequency of backup shipping using cron syntax. You also need to set options that name the schedule and define various aspects of the backup shipping, such as on-failure actions, the number of local and remote backup copies to keep, and whether to also schedule incremental backup shipping.

At a minimum, the command needs a schedule name and a full backup cron schema to create a backup shipping schedule. An example command would look like this:

# linstor schedule create \
  --incremental-cron '* * * * *' \ (1)
  --keep-local 5 \ (2)
  --keep-remote 4 \ (3)
  --on-failure RETRY \ (4)
  --max-retries 10 \ (5)
  <schedule_name> \ (6)
  '* * * * *' # full backup cron schema (7)
Enclose cron schemas within single or double quotation marks.
1 If specified, the incremental cron schema describes how frequently to create and ship incremental backups. New incremental backups are based on the most recent full backup.
2 The --keep-local option allows you to specify how many snapshots that a full backup is based upon should be kept at the local backup source. If unspecified, all snapshots will be kept. [OPTIONAL]
3 The --keep-remote option allows you to specify how many full backups should be kept at the remote destination. This option only works with S3 remote backup destinations, because you would not want to allow a cluster node to delete backups from a node in another cluster. All incremental backups based on a deleted full backup will also be deleted at the remote destination. If unspecified, the --keep-remote option defaults to “all”. [OPTIONAL]
4 Specifies whether to “RETRY” or “SKIP” the scheduled backup shipping if it fails. If “SKIP” is specified, LINSTOR will ignore the failure and continue with the next scheduled backup shipping. If “RETRY” is specified, LINSTOR will wait 60 seconds and then try the backup shipping again. The LINSTOR schedule create command defaults to “SKIP” if no --on-failure option is given. [OPTIONAL]
5 The number of times to retry the backup shipping if a scheduled backup shipping fails and the --on-failure RETRY option has been given. Without this option, the LINSTOR controller will retry the scheduled backup shipping indefinitely, until it is successful. [OPTIONAL]
6 The name that you give the backup schedule so that you can reference it later with the schedule list, modify, delete, enable, or disable commands. [REQUIRED]
7 This cron schema describes how frequently LINSTOR creates snapshots and ships full backups.
If you specify an incremental cron schema that has overlap with the full cron schema that you specify, at the times when both types of backup shipping would occur simultaneously, LINSTOR will only make and ship a full backup. For example, if you specify that a full backup be made every three hours, and an incremental backup be made every hour, then every third hour, LINSTOR will only make and ship a full backup. For this reason, specifying the same cron schema for both your incremental and full backup shipping schedules would be useless, because incremental backups will never be made.

2.12.2. Modifying a Backup Shipping Schedule

You can modify a backup shipping schedule by using the LINSTOR client schedule modify command. The syntax for the command is the same as that for the schedule create command. The name that you specify with the schedule modify command must be an already existing backup schedule. Any options to the command that you do not specify will retain their existing values. If you want to set the keep-local or keep-remote options back to their default values, you can set them to “all”. If you want to set the max-retries option to its default value, you can set it to “forever”.

2.12.3. Configuring the Number of Local Snapshots and Remote Backups to Keep

Your physical storage is not infinite and your remote storage has a cost, so you will likely want to set limits on the number of snapshots and backups you keep.

Both the --keep-remote and --keep-local options deserve special mention as they have implications beyond what may be obvious. Using these options, you specify how many snapshots or full backups should be kept, either on the local source or the remote destination.

Configuring the Keep-local Option

For example, if a --keep-local=2 option is set, then the backup shipping schedule, on first run, will make a snapshot for a full backup. On the next scheduled full backup shipping, it will make a second snapshot for a full backup. On the next scheduled full backup shipping, it makes a third snapshot for a full backup. This time, however, after successful completion, LINSTOR deletes the first (oldest) full backup shipping snapshot. If snapshots were made for any incremental backups based on this full snapshot, they will also be deleted from the local source node. On the next successful full backup shipping, LINSTOR will delete the second full backup snapshot and any incremental snapshots based upon it, and so on, with each successive backup shipping.

If there are local snapshots remaining from failed shipments, these will be deleted first, even if they were created later.

If you have enabled a backup shipping schedule and then later manually delete a LINSTOR snapshot, LINSTOR may not be able to delete everything it was supposed to. For example, if you delete a full backup snapshot definition, on a later full backup scheduled shipping, there may be incremental snapshots based on the manually deleted full backup snapshot that will not be deleted.

Configuring the Keep-remote Option

As mentioned in the callouts for the example linstor schedule create command above, the keep-remote option only works for S3 remote destinations. Here is an example of how the option works. If a --keep-remote=2 option is set, then the backup shipping schedule, on first run, will make a snapshot for a full backup and ship it to the remote destination. On the next scheduled full backup shipping, a second snapshot is made and a full backup shipped to the remote destination. On the next scheduled full backup shipping, a third snapshot is made and a full backup shipped to the remote destination. This time, additionally, after the third snapshot successfully ships, the first full backup is deleted from the remote destination. If any incremental backups were scheduled and made between the full backups, any that were made from the first full backup would be deleted along with the full backup.

This option only deletes backups at the remote destination. It does not delete snapshots that the full backups were based upon at the local source node.

2.12.4. Listing a Backup Shipping Schedule

You can list your backup shipping schedules by using the linstor schedule list command.

For example:

# linstor schedule list
╭──────────────────────────────────────────────────────────────────────────────────────╮
┊ Name                ┊ Full        ┊ Incremental ┊ KeepLocal ┊ KeepRemote ┊ OnFailure ┊
╞══════════════════════════════════════════════════════════════════════════════════════╡
┊ my-bu-schedule      ┊ 2 * * * *   ┊             ┊ 3         ┊ 2          ┊ SKIP      ┊
╰──────────────────────────────────────────────────────────────────────────────────────╯

2.12.5. Deleting a Backup Shipping Schedule

The LINSTOR client schedule delete command completely deletes a backup shipping schedule LINSTOR object. The command’s only argument is the schedule name that you want to delete. If the deleted schedule is currently creating or shipping a backup, the scheduled shipping process is stopped. Depending on at which point the process stops, a snapshot, or a backup, or both, might not be created and shipped.

This command does not affect previously created snapshots or successfully shipped backups. These will be retained until they are manually deleted.

2.12.6. Enabling Scheduled Backup Shipping

You can use the LINSTOR client backup schedule enable command to enable a previously created backup shipping schedule. The command has the following syntax:

# linstor backup schedule enable \
  [--node source_node] \ (1)
  [--rg resource_group_name | --rd resource_definition_name] \ (2)
  remote_name \ (3)
  schedule_name (4)
1 This is a special option that allows you to specify the controller node that will be used as a source for scheduled backup shipments, if possible. If you omit this option from the command, then LINSTOR will choose a source node at the time a scheduled shipping is made. [OPTIONAL]
2 You can set here either the resource group or the resource definition (but not both) that you want to enable the backup shipping schedule for. If you omit this option from the command, then the command enables scheduled backup shipping for all deployed LINSTOR resources that can make snapshots. [OPTIONAL]
3 The name of the remote destination that you want to ship backups to. [REQUIRED]
4 The name of a previously created backup shipping schedule. [REQUIRED]

2.12.7. Disabling a Backup Shipping Schedule

To disable a previously enabled backup shipping schedule, you use the LINSTOR client backup schedule disable command. The command has the following syntax:

# linstor backup schedule disable \
  [--rg resource_group_name | --rd resource_definition_name] \
  remote_name \ (3)
  schedule_name (4)

If you include the option specifying either a resource group or resource definition, as described in the backup schedule enable command example above, then you disable the schedule only for that resource group or resource definition.

For example, if you omitted specifying a resource group or resource definition in an earlier backup schedule enable command, LINSTOR would schedule backup shipping for all its deployed resources that can make snapshots. Your disable command would then only affect the resource group or resource definition that you specify with the command. The backup shipping schedule would still apply to any deployed LINSTOR resources besides the specified resource group or resource definition.

The same as for the backup schedule enable command, if you specify neither a resource group nor a resource definition, then LINSTOR disables the backup shipping schedule at the controller level for all deployed LINSTOR resources.

2.12.8. Deleting Aspects of a Backup Shipping Schedule

You can use the linstor backup schedule delete command to granularly delete either a specified resource definition or a resource group from a backup shipping schedule, without deleting the schedule itself. This command has the same syntax and arguments as the backup schedule disable command. If you specify neither a resource group nor a resource definition, the backup shipping schedule you specify will be deleted at the controller level.

It may be helpful to think about the backup schedule delete command as a way that you can remove a backup shipping schedule-remote pair from a specified LINSTOR object level, either a resource definition, a resource group, or at the controller level if neither is specified.

The backup schedule delete command does not affect previously created snapshots or successfully shipped backups. These will be retained until they are manually deleted, or until they are removed by the effects of a still applicable keep-local or keep-remote option.

You might want to use this command when you have disabled a backup schedule for multiple LINSTOR object levels and later want to affect a granular change, where a backup schedule enable command might have unintended consequences.

For example, consider a scenario where you have a backup schedule-remote pair that you enabled at a controller level. This controller has a resource group, myresgroup that has several resource definitions, resdef1 through resdef9, under it. For maintenance reasons perhaps, you disable the schedule for two resource definitions, resdef1 and resdef2. You then realize that further maintenance requires that you disable the backup shipping schedule at the resource group level, for your myresgroup resource group.

After completing some maintenance, you are able to enable the backup shipping schedule for resdef3 through resdef9, but you are not yet ready to resume (enable) backup shipping for resdef1 and resdef2. You can enable backup shipping for each resource definition individually, resdef3 through resdef9, or you can use the backup schedule delete command to delete the backup shipping schedule from the resource group, myresgroup. If you use the backup schedule delete command, backups of resdef3 through resdef9 will ship again because the backup shipping schedule is enabled at the controller level, but resdef1 and resdef2 will not ship because the backup shipping schedule is still disabled for them at the resource definition level.

When you complete your maintenance and are again ready to ship backups for resdef1 and resdef2, you can delete the backup shipping schedule for those two resource definitions to return to your starting state: backup shipping scheduled for all LINSTOR deployed resources at the controller level. To visualize this it may be helpful to refer to the decision tree diagram for how LINSTOR decides whether or not to ship a backup in the How the LINSTOR Controller Determines Scheduled Backup Shipping subsection.

In the example scenario above, you might have enabled backup shipping on the resource group, after completing some maintenance. In this case, backup shipping would resume for resource definitions resdef3 through resdef9 but continue not to ship for resource definitions resdef1 and resdef2 because backup shipping was still disabled for those resource definitions. After you completed all maintenance, you could delete the backup shipping schedule on resdef1 and resdef2. Then all of your resource definitions would be shipping backups, as they were prior to your maintenance, because the schedule-remote pair was enabled at the resource group level. However, this would remove your option to globally stop all scheduled shipping at some later point in time at the controller level because the enabled schedule at the resource group level would override any schedule disable command applied at the controller level.

2.12.9. Listing Backup Shipping Schedules by Resource

You can list backup schedules by resource, using the LINSTOR client schedule list-by-resource command. This command will show LINSTOR resources and how any backup shipping schedules apply and to which remotes they are being shipped. If resources are not being shipped then the command will show:

  • Whether resources have no schedule-remote-pair entries (empty cells)

  • Whether they have schedule-remote-pair entries but they are disabled (“disabled”)

  • Whether they have no resources, so no backup shipments can be made, regardless of whether any schedule-remote-pair entries are enabled or not (“undeployed”)

If resources have schedule-remote-pairs and are being shipped, the command output will show when the last backup was shipped and when the next backup is scheduled to ship. It will also show whether the next and last backup shipments were full or incremental backups. Finally, the command will show when the next planned incremental (if any) and full backup shipping will occur.

You can use the --active-only flag with the schedule list-by-resource command to filter out all resources that are not being shipped.

2.12.10. How the LINSTOR Controller Determines Scheduled Backup Shipping

To determine if the LINSTOR Controller will ship a deployed LINSTOR resource with a certain backup schedule for a given remote destination, the LINSTOR Controller uses the following logic:

linstor controller backup schedule decision flowchart

As the diagram shows, enabled or disabled backup shipping schedules have effect in the following order:

  1. Resource definition level

  2. Resource group level

  3. Controller level

A backup shipping schedule-remote pair that is enabled or disabled at a preceding level will override the enabled or disabled status for the same schedule-remote pair at a later level.

2.12.11. Determining How Scheduled Backup Shipping Affects a Resource

To determine how a LINSTOR resource will be affected by scheduled backup shipping, you can use the LINSTOR client schedule list-by-resource-details command for a specified LINSTOR resource.

The command will output a table that shows on what LINSTOR object level a backup shipping schedule is either not set (empty cell), enabled, or disabled.

By using this command, you can determine on which level you need to make a change to enable, disable, or delete scheduled backup shipping for a resource.

Example output could look like this:

# linstor schedule list-by-resource-details my_linstor_resource_name
╭───────────────────────────────────────────────────────────────────────────╮
┊ Remote   ┊ Schedule   ┊ Resource-Definition ┊ Resource-Group ┊ Controller ┊
╞═══════════════════════════════════════════════════════════════════════════╡
┊ rem1     ┊ sch1       ┊ Disabled            ┊                ┊ Enabled    ┊
┊ rem1     ┊ sch2       ┊                     ┊ Enabled        ┊            ┊
┊ rem2     ┊ sch1       ┊ Enabled             ┊                ┊            ┊
┊ rem2     ┊ sch5       ┊                     ┊ Enabled        ┊            ┊
┊ rem3     ┊ sch4       ┊                     ┊ Disabled       ┊ Enabled    ┊
╰───────────────────────────────────────────────────────────────────────────╯

2.13. Setting DRBD Options for LINSTOR Objects

You can use LINSTOR commands to set DRBD options. Configurations in files that are not managed by LINSTOR, such as /etc/drbd.d/global_common.conf, will be ignored. The syntax for this command is generally:

# linstor <LINSTOR_object> drbd-options --<DRBD_option> <value> <LINSTOR_object_identifiers>

In the syntax above, <LINSTOR_ object_identifiers> is a placeholder for identifiers such as a node name, node names, or a resource name, or a combination of these identifiers.

For example, to set the DRBD replication protocol for a resource definition named backups, enter:

# linstor resource-definition drbd-options --protocol C backups

You can enter a LINSTOR object along with drbd-options and the --help, or -h, flag to show the command usage, available options, and the default value for each option. For example:

# linstor controller drbd-options -h

2.13.1. Setting DRBD Peer Options for LINSTOR Resources or Resource Connections

The LINSTOR resource object is an exception to the general syntax for setting DRBD options for LINSTOR objects. With the LINSTOR resource object, you can use the drbd-peer-options to set DRBD options at the connection level between the two nodes that you specify. Specifying drbd-peer-options for a LINSTOR resource object between two nodes is equivalent to using the`linstor resource-connection drbd-peer-options` for a resource between two nodes.

For example, to set the DRBD maximum buffer size to 8192 at a connection level, for a resource named backups, between two nodes, node-0 and node-1, enter:

# linstor resource drbd-peer-options --max-buffers 8192 node-0 node-1 backups

The command above is equivalent to the following:

# linstor resource-connection drbd-peer-options --max-buffers 8192 node-0 node-1 backups

Indeed, when using the linstor --curl command to examine the two commands actions on the LINSTOR REST API, the output is identical:

# linstor --curl resource drbd-peer-options --max-buffers 8192 node-0 node-1 backups
curl -X PUT -H "Content-Type: application/json" -d '{"override_props": {"DrbdOptions/Net/max-buffers": "8192"}}' http://localhost:3370/v1/resource-definitions/backups/resource-connections/node-0/node-1

# linstor --curl resource-connection drbd-peer-options --max-buffers 8192 node-0 node-1 backups
curl -X PUT -H "Content-Type: application/json" -d '{"override_props": {"DrbdOptions/Net/max-buffers": "8192"}}' http://localhost:3370/v1/resource-definitions/backups/resource-connections/node-0/node-1

The connection section of the LINSTOR-generated resource file backups.res on node-0 will look something like this:

connection {
        _peer_node_id 1;
        path {
            _this_host ipv4 192.168.222.10:7000;
            _remote_host ipv4 192.168.222.11:7000;
        }
        path {
            _this_host ipv4 192.168.121.46:7000;
            _remote_host ipv4 192.168.121.220:7000;
        }
        net {
			[...]
            max-buffers         8192;
            _name               "node-1";
        }
    }
If there are multiple paths between the two nodes, as in the example above, DRBD options that you set using the resource drbd-peer-options command will apply to all of them.

2.13.2. Setting DRBD Options for Node Connections

You can use the drbd-peer-options argument to set DRBD options at a connection level, between two nodes, for example:

# linstor node-connection drbd-peer-options --ping-timeout 299 node-0 node-1

The preceding command would set the DRBD ping-timeout option to 29.9 seconds at a connection level between two nodes, node-0 and node-1.

2.13.3. Verifying Options for LINSTOR Objects

You can verify a LINSTOR object’s set properties by using the list-properties command, for example:

|======================================================|
# linstor resource-definition list-properties backups
+------------------------------------------------------+
| Key                               | Value            |
| DrbdOptions/Net/protocol          | C                |
[...]
  [[s-linstor-unset-opts]]
==== Removing DRBD Options from LINSTOR Objects

To remove a previously set DRBD option, prefix the option name with unset-. For example:

# linstor resource-definition drbd-options --unset-protocol backups

The same syntax applies to any drbd-peer-options set either on a LINSTOR resource, resource connection, or node connection. For example:

# linstor resource-connection drbd-peer-options --unset-max-buffers node-0 node-1 backups
  Removing a DRBD option or DRBD peer option will return the option to its
default value. Refer to the `linstor <LINSTOR_object> drbd-options --help`
(or `drbd-peer-options --help`) command output for the default values of
options. You can also refer to the `drbd.conf-9.0` man page to get
information about DRBD options.  [[s-linstor-toggle-disk]]
=== Adding and Removing Disks

LINSTOR can convert resources between diskless and having a disk. This is achieved with the resource toggle-disk command, which has syntax similar to resource create.

For instance, add a disk to the diskless resource backups on ‘alpha’:

# linstor resource toggle-disk alpha backups --storage-pool pool_ssd

Remove this disk again:

# linstor resource toggle-disk alpha backups --diskless
  [[s-linstor-migrate-disk]]
==== Migrating Disks Between Nodes

To move a resource between nodes without reducing redundancy at any point, LINSTOR’s disk migrate feature can be used. First create a diskless resource on the target node, and then add a disk using the --migrate-from option. This will wait until the data has been synced to the new disk and then remove the source disk.

For example, to migrate a resource backups from ‘alpha’ to ‘bravo’:

# linstor resource create bravo backups --drbd-diskless
# linstor resource toggle-disk bravo backups --storage-pool pool_ssd --migrate-from alpha
  [[s-linstor-proxy]]
=== Configuring DRBD Proxy Using LINSTOR

LINSTOR expects DRBD Proxy to be running on the nodes which are involved in the relevant connections. It does not currently support connections through DRBD Proxy on a separate node.

Suppose our cluster consists of nodes ‘alpha’ and ‘bravo’ in a local network and ‘charlie’ at a remote site, with a resource definition named backups deployed to each of the nodes. Then DRBD Proxy can be enabled for the connections to ‘charlie’ as follows:

# linstor drbd-proxy enable alpha charlie backups
# linstor drbd-proxy enable bravo charlie backups

The DRBD Proxy configuration can be tailored with commands such as:

# linstor drbd-proxy options backups --memlimit 100000000
# linstor drbd-proxy compression zlib backups --level 9

LINSTOR does not automatically optimize the DRBD configuration for long-distance replication, so you will probably want to set some configuration options such as the protocol:

# linstor resource-connection drbd-options alpha charlie backups --protocol A
# linstor resource-connection drbd-options bravo charlie backups --protocol A
Please contact LINBIT for assistance optimizing your configuration.

2.13.4. Automatically Enabling DRBD Proxy

LINSTOR can also be configured to automatically enable the above mentioned Proxy connection between two nodes. For this automation, LINSTOR first needs to know on which site each node is.

# linstor node set-property alpha Site A
# linstor node set-property bravo Site A
# linstor node set-property charlie Site B

As the Site property might also be used for other site-based decisions in future features, the DrbdProxy/AutoEnable also has to be set to true:

# linstor controller set-property DrbdProxy/AutoEnable true
  This property can also be set on node, resource-definition, resource and
resource-connection level (from left to right in increasing priority,
whereas the controller is the left-most, that is, the least prioritized
level).  Once this initialization steps are completed, every newly created
resource will automatically check if it has to enable DRBD proxy to any of
its peer-resources.  [[s-linstor-external-database]]
=== External Database Providers
  It is possible to have LINSTOR working with an external database provider
like PostgreSQL, MariaDB and since version 1.1.0 even etcd key value store
is supported.  To use an external database there are a few additional steps
to configure.  You have to create a DB/Schema and user to use for linstor,
and configure this in the `/etc/linstor/linstor.toml`.  [[s-postgresql]]
==== PostgreSQL

A sample PostgreSQL linstor.toml looks like this:

[db]
user = "linstor"
password = "linstor"
connection_url = "jdbc:postgresql://localhost/linstor"
  [[s-mariadb_mysql]]
==== MariaDB and MySQL

A sample MariaDB linstor.toml looks like this:

[db]
user = "linstor"
password = "linstor"
connection_url = "jdbc:mariadb://localhost/LINSTOR?createDatabaseIfNotExist=true"
  NOTE: The LINSTOR schema/database is created as `LINSTOR` so verify that
the MariaDB connection string refers to the `LINSTOR` schema, as in the
example above.  [[s-etcd]]
==== etcd

etcd is a distributed key-value store that makes it easy to keep your LINSTOR database distributed in a HA-setup. The etcd driver is already included in the LINSTOR-controller package and only needs to be configured in the linstor.toml.

More information about how to install and configure etcd can be found here: etcd docs

And here is a sample [db] section from the linstor.toml:

[db]
## only set user/password if you want to use authentication, only since LINSTOR 1.2.1
# user = "linstor"
# password = "linstor"

## for etcd
## do not set user field if no authentication required
connection_url = "etcd://etcdhost1:2379,etcdhost2:2379,etcdhost3:2379"

## if you want to use TLS, only since LINSTOR 1.2.1
# ca_certificate = "ca.pem"
# client_certificate = "client.pem"

## if you want to use client TLS authentication too, only since LINSTOR 1.2.1
# client_key_pkcs8_pem = "client-key.pkcs8"
## set client_key_password if private key has a password
# client_key_password = "mysecret"
  [[s-linstor-controller-config]]
=== Configuring the LINSTOR Controller
  The LINSTOR Controller has a configuration file that is and has to be
placed into the following path: `/etc/linstor/linstor.toml`.  A recent
configuration example can be found here:
https://github.com/LINBIT/linstor-server/blob/master/docs/linstor.toml-example[linstor.toml-example]

2.13.5. LINSTOR REST API

To make LINSTOR’s administrative tasks more accessible and also available for web-frontends a REST API has been created. The REST API is embedded in the linstor-controller and since LINSTOR 0.9.13 configured through the linstor.toml configuration file.

[http]
  enabled = true
  port = 3370
  listen_addr = "127.0.0.1"  # to disable remote access
  If you want to use the REST API the current documentation can be found on
the following link: https://app.swaggerhub.com/apis-docs/Linstor/Linstor/

2.13.6. LINSTOR REST API HTTPS

The HTTP REST API can also run secured by HTTPS and is highly recommended if you use any features that require authorization. To do so you have to create a Java keystore file with a valid certificate that will be used to encrypt all HTTPS traffic.

Here is a simple example on how you can create a self signed certificate with the keytool that is included in the Java Runtime:

keytool -keyalg rsa -keysize 2048 -genkey -keystore ./keystore_linstor.jks\
 -alias linstor_controller\
 -dname "CN=localhost, OU=SecureUnit, O=ExampleOrg, L=Vienna, ST=Austria, C=AT"

keytool will ask for a password to secure the generated keystore file and is needed for the LINSTOR Controller configuration. In your linstor.toml file you have to add the following section:

[https]
  keystore = "/path/to/keystore_linstor.jks"
  keystore_password = "linstor"
  Now (re)start the `linstor-controller` and the HTTPS REST API should be
available on port 3371.  More information about how to import other
certificates can be found here:
https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html
NOTE: When HTTPS is enabled, all requests to the HTTP /v1/ REST API will be
redirected to the HTTPS redirect.  [[s-rest-api-https-restricted-client]]
===== LINSTOR REST API HTTPS Restricted Client Access

Client access can be restricted by using a SSL/TLS truststore on the Controller. Basically you create a certificate for your client and add it to your truststore and the client then uses this certificate for authentication.

First create a client certificate:

keytool -keyalg rsa -keysize 2048 -genkey -keystore client.jks\
 -storepass linstor -keypass linstor\
 -alias client1\
 -dname "CN=Client Cert, OU=client, O=Example, L=Vienna, ST=Austria, C=AT"

Then we import this certificate to our controller truststore:

keytool -importkeystore\
 -srcstorepass linstor -deststorepass linstor -keypass linstor\
 -srckeystore client.jks -destkeystore trustore_client.jks

And enable the truststore in the linstor.toml configuration file:

[https]
  keystore = "/path/to/keystore_linstor.jks"
  keystore_password = "linstor"
  truststore = "/path/to/trustore_client.jks"
  truststore_password = "linstor"

Now restart the Controller and it will no longer be possible to access the controller API without a correct certificate.

The LINSTOR client needs the certificate in PEM format, so before we can use it we have to convert the java keystore certificate to the PEM format.

# Convert to pkcs12
keytool -importkeystore -srckeystore client.jks -destkeystore client.p12\
 -storepass linstor -keypass linstor\
 -srcalias client1 -srcstoretype jks -deststoretype pkcs12

# use openssl to convert to PEM
openssl pkcs12 -in client.p12 -out client_with_pass.pem

To avoid entering the PEM file password all the time it might be convenient to remove the password.

openssl rsa -in client_with_pass.pem -out client1.pem
openssl x509 -in client_with_pass.pem >> client1.pem

Now this PEM file can easily be used in the client:

linstor --certfile client1.pem node list
  The `--certfile` parameter can also added to the client configuration
file, see <<s-using_the_linstor_client>> for more details.

2.14. Configuring LINSTOR Satellite

  The LINSTOR Satellite software has an optional configuration file that
uses the TOML file syntax and has to be put into the following path
`/etc/linstor/linstor_satellite.toml`.  A recent configuration example can
be found here:
https://github.com/LINBIT/linstor-server/blob/master/docs/linstor_satellite.toml-example[linstor_satellite.toml-example]

2.15. Logging

LINSTOR uses SLF4J with Logback as binding. This gives LINSTOR the possibility to distinguish between the log levels ERROR, WARN, INFO, DEBUG and TRACE (in order of increasing verbosity). In the current linstor version (1.1.2) the user has the following four methods to control the logging level, ordered by priority (first has highest priority):

  1. TRACE mode can be enabled or disabled using the debug console:

    Command ==> SetTrcMode MODE(enabled)
    SetTrcMode           Set TRACE level logging mode
    New TRACE level logging mode: ENABLED
  2. When starting the controller or satellite a command line argument can be passed:

    java ... com.linbit.linstor.core.Controller ... --log-level TRACE
    java ... com.linbit.linstor.core.Satellite  ... --log-level TRACE
  3. The recommended place is the logging section in the configuration file. The default configuration file location is /etc/linstor/linstor.toml for the controller and /etc/linstor/linstor_satellite.toml for the satellite. Configure the logging level as follows:

    [logging]
       level="TRACE"
  4. As LINSTOR is using Logback as an implementation, /usr/share/linstor-server/lib/logback.xml can also be used. Currently only this approach supports different log levels for different components, like shown in the example below:

    <?xml version="1.0" encoding="UTF-8"?>
    <configuration scan="false" scanPeriod="60 seconds">
    <!--
     Values for scanPeriod can be specified in units of milliseconds, seconds, minutes or hours
     https://logback.qos.ch/manual/configuration.html
    -->
     <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
       <!-- encoders are assigned the type
            ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
       <encoder>
         <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger - %msg%n</pattern>
       </encoder>
     </appender>
     <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
       <file>${log.directory}/linstor-${log.module}.log</file>
       <append>true</append>
       <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
         <Pattern>%d{yyyy_MM_dd HH:mm:ss.SSS} [%thread] %-5level %logger - %msg%n</Pattern>
       </encoder>
       <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
         <FileNamePattern>logs/linstor-${log.module}.%i.log.zip</FileNamePattern>
         <MinIndex>1</MinIndex>
         <MaxIndex>10</MaxIndex>
       </rollingPolicy>
       <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
         <MaxFileSize>2MB</MaxFileSize>
       </triggeringPolicy>
     </appender>
     <logger name="LINSTOR/Controller" level="TRACE" additivity="false">
       <appender-ref ref="STDOUT" />
       <!-- <appender-ref ref="FILE" /> -->
     </logger>
     <logger name="LINSTOR/Satellite" level="TRACE" additivity="false">
       <appender-ref ref="STDOUT" />
       <!-- <appender-ref ref="FILE" /> -->
     </logger>
     <root level="WARN">
       <appender-ref ref="STDOUT" />
       <!-- <appender-ref ref="FILE" /> -->
     </root>
    </configuration>
      See the https://logback.qos.ch/manual/index.html[Logback Manual] to find
    more details about `logback.xml`.  When none of the configuration methods
    above is used LINSTOR will default to `INFO` log level.

    === Monitoring

Since LINSTOR 1.8.0, a Prometheus /metrics HTTP path is provided with LINSTOR and JVM specific exports.

The /metrics path also supports three GET arguments to reduce LINSTOR’s reported data:

  • resource

  • storage_pools

  • error_reports

These all default to true. To disable, for example error report data: http://localhost:3370/metrics?error_reports=false

2.15.1. Health Checking

  The LINSTOR-Controller also provides a `/health` HTTP path that will
simply return HTTP-Status 200 if the controller can access its database and
all services are up and running. Otherwise it will return HTTP error status
code 500 `Internal Server Error`.  [[s-linstor-secure-connections]]
=== Secure Satellite Connections

It is possible to have the LINSTOR use SSL/TLS secure TCP connection between controller and satellite connections. Without going into further details on how Java’s SSL/TLS engine works we will give you command line snippets using the keytool from Java’s runtime environment on how to configure a three node setup using secure connections.

The node setup looks like this:

Node alpha is the just the controller. Node bravo and node charlie are just satellites.

Here are the commands to generate such a keystore setup, values should of course be edited for your environment.

# create directories to hold the key files
mkdir -p /tmp/linstor-ssl
cd /tmp/linstor-ssl
mkdir alpha bravo charlie


# create private keys for all nodes
keytool -keyalg rsa -keysize 2048 -genkey -keystore alpha/keystore.jks\
 -storepass linstor -keypass linstor\
 -alias alpha\
 -dname "CN=Max Mustermann, OU=alpha, O=Example, L=Vienna, ST=Austria, C=AT"

keytool -keyalg rsa -keysize 2048 -genkey -keystore bravo/keystore.jks\
 -storepass linstor -keypass linstor\
 -alias bravo\
 -dname "CN=Max Mustermann, OU=bravo, O=Example, L=Vienna, ST=Austria, C=AT"

keytool -keyalg rsa -keysize 2048 -genkey -keystore charlie/keystore.jks\
 -storepass linstor -keypass linstor\
 -alias charlie\
 -dname "CN=Max Mustermann, OU=charlie, O=Example, L=Vienna, ST=Austria, C=AT"

# import truststore certificates for alpha (needs all satellite certificates)
keytool -importkeystore\
 -srcstorepass linstor -deststorepass linstor -keypass linstor\
 -srckeystore bravo/keystore.jks -destkeystore alpha/certificates.jks

keytool -importkeystore\
 -srcstorepass linstor -deststorepass linstor -keypass linstor\
 -srckeystore charlie/keystore.jks -destkeystore alpha/certificates.jks

# import controller certificate into satellite truststores
keytool -importkeystore\
 -srcstorepass linstor -deststorepass linstor -keypass linstor\
 -srckeystore alpha/keystore.jks -destkeystore bravo/certificates.jks

keytool -importkeystore\
 -srcstorepass linstor -deststorepass linstor -keypass linstor\
 -srckeystore alpha/keystore.jks -destkeystore charlie/certificates.jks

# now copy the keystore files to their host destinations
ssh [email protected] mkdir /etc/linstor/ssl
scp alpha/* [email protected]:/etc/linstor/ssl/
ssh [email protected] mkdir /etc/linstor/ssl
scp bravo/* [email protected]:/etc/linstor/ssl/
ssh [email protected] mkdir /etc/linstor/ssl
scp charlie/* [email protected]:/etc/linstor/ssl/

# generate the satellite ssl config entry
echo '[netcom]
  type="ssl"
  port=3367
  server_certificate="ssl/keystore.jks"
  trusted_certificates="ssl/certificates.jks"
  key_password="linstor"
  keystore_password="linstor"
  truststore_password="linstor"
  ssl_protocol="TLSv1.2"
' | ssh [email protected] "cat > /etc/linstor/linstor_satellite.toml"

echo '[netcom]
  type="ssl"
  port=3367
  server_certificate="ssl/keystore.jks"
  trusted_certificates="ssl/certificates.jks"
  key_password="linstor"
  keystore_password="linstor"
  truststore_password="linstor"
  ssl_protocol="TLSv1.2"
' | ssh [email protected] "cat > /etc/linstor/linstor_satellite.toml"
  Now just start controller and satellites and add the nodes with
`--communication-type SSL`.  [[s-linstor-ldap-authentication]]
=== Configuring LDAP Authentication

You can configure LINSTOR to use LDAP authentication to limit access to the LINSTOR Controller. This feature is disabled by default but you can enable and configure it by editing the LINSTOR configuration TOML file. After editing the configuration file, you will need to restart the linstor-controller.service. An example LDAP section within the configuration file looks like this:

[ldap]
  enabled = true (1)

  # allow_public_access: if no authorization fields are given allow
  # users to work with the public context
  allow_public_access = false (2)

  # uniform resource identifier: LDAP URI to use
  # for example, "ldaps://hostname" (LDAPS) or "ldap://hostname" (LDAP)
  uri = "ldaps://ldap.example.com"

  # distinguished name: {user} can be used as template for the user name
  dn = "uid={user}" (3)

  # search base for the search_filter field
  search_base = "dc=example,dc=com" (4)

  # search_filter: ldap filter to restrict users on memberships
  search_filter = "(&(uid={user})(memberof=ou=storage-services,dc=example,dc=com))" (5)
1 enabled is a Boolean value. Authentication is disabled by default.
2 allow_public_access is a Boolean value. If set to true, and LDAP authentication is enabled, then users will be allowed to work with the LINSTOR Controller’s public context. If set to false and LDAP authentication is enabled, then users without LDAP authenticating credentials will be unable to access the LINSTOR Controller for all but the most trivial tasks, such as displaying version or help information.
3 dn is a string value where you can specify the LDAP distiguished name to query the LDAP directory. Besides the user ID (uid), the string may consist of other distinguished name attributes, for example:
dn = "uid={user},ou=storage-services,o=ha,dc=example"
4 search_base is a string value where you can specify the starting point in the LDAP directory tree for the authentication query, for example:
search_base = "ou=storage-services"
5 search_filter is a string value where you can specify an LDAP object restriction for authentication, such as user and group membership, for example:
search_filter = "(&(uid={user})(memberof=ou=storage-services,dc=example,dc=com))"
  WARNING: It is highly recommended that you configure
<<s-linstor-rest-api-https,LINSTOR REST API HTTPS>> and LDAPS to protect
potentially sensitive traffic passing between the LINSTOR Controller and an
LDAP server.

2.15.2. Running LINSTOR Commands as an Authenticated User

After configuring the LINSTOR Controller to authenticate users through LDAP (or LDAPS), and the LINSTOR REST API HTTPS, you will need to enter LINSTOR commands as follows:

$ linstor --user <LDAP_user_name> <command>

If you have configured LDAP authentication without also configuring LINSTOR REST API HTTPS, you will need to explicitly enable password authentication over HTTP, by using the --allow-insecure-path flag with your linstor commands. This is not recommended outside of a secured and isolated LAN, as you will be sending credentials in plain text.

$ linstor --allow-insecure-auth --user <LDAP_user_name> <command>
  The LINSTOR Controller will prompt you for the user's password, in each of
the above examples. You may optionally use the `--password` argument to
supply the user's password on the command line, with all the warnings of
caution that would go along with doing so.  [[s-linstor-drbd-automatisms]]
=== Automatisms for DRBD Resources
  This section details some of LINSTOR's automatisms for DRBD resources.

2.15.3. Auto-Quorum Policies

LINSTOR automatically configures quorum policies on resources when quorum is achievable. This means, whenever you have at least two diskful and one or more diskless resource assignments, or three or more diskful resource assignments, LINSTOR will enable quorum policies for your resources automatically.

Inversely, LINSTOR will automatically disable quorum policies whenever there are less than the minimum required resource assignments to achieve quorum.

This is controlled through the, DrbdOptions/auto-quorum, property which can be applied to the linstor-controller, resource-group, and resource-definition. Accepted values for the DrbdOptions/auto-quorum property are disabled, suspend-io, and io-error.

Setting the DrbdOptions/auto-quorum property to disabled will allow you to manually, or more granularly, control the quorum policies of your resources should you want to.

The default policies for DrbdOptions/auto-quorum are quorum majority, and on-no-quorum io-error. For more information about DRBD’s quorum features and their behavior, please refer to the quorum section of the DRBD user’s guide.
The DrbdOptions/auto-quorum policies will override any manually configured properties if DrbdOptions/auto-quorum is not disabled.

For example, to manually set the quorum policies of a resource-group named my_ssd_group, you would use the following commands:

# linstor resource-group set-property my_ssd_group DrbdOptions/auto-quorum disabled
# linstor resource-group set-property my_ssd_group DrbdOptions/Resource/quorum majority
# linstor resource-group set-property my_ssd_group DrbdOptions/Resource/on-no-quorum suspend-io

You may want to disable DRBD’s quorum features completely. To do that, you would need to first disable DrbdOptions/auto-quorum on the appropriate LINSTOR object, and then set the DRBD quorum features accordingly. For example, use the following commands to disable quorum entirely on the my_ssd_group resource-group:

# linstor resource-group set-property my_ssd_group DrbdOptions/auto-quorum disabled
# linstor resource-group set-property my_ssd_group DrbdOptions/Resource/quorum off
# linstor resource-group set-property my_ssd_group DrbdOptions/Resource/on-no-quorum
  NOTE: Setting `DrbdOptions/Resource/on-no-quorum` to an empty value in the
commands above deletes the property from the object entirely.

2.15.4. Auto-Evict

If a satellite is offline for a prolonged period of time, LINSTOR can be configured to declare that node as evicted. This triggers an automated reassignment of the affected DRBD resources to other nodes to ensure a minimum replica count is kept.

This feature uses the following properties to adapt the behaviour.

  • DrbdOptions/AutoEvictMinReplicaCount sets the number of replicas that should always be present. You can set this property on the controller to change a global default, or on a specific resource-definition or resource-group to change it only for that resource-definition or resource-group. If this property is left empty, the place-count set for the auto-placer of the corresponding resource-group will be used.

  • DrbdOptions/AutoEvictAfterTime describes how long a node can be offline in minutes before the eviction is triggered. You can set this property on the controller to change a global default, or on a single node to give it a different behavior. The default value for this property is 60 minutes.

  • DrbdOptions/AutoEvictMaxDisconnectedNodes sets the percentage of nodes that can be not reachable (for whatever reason) at the same time. If more than the given percent of nodes are offline at the same time, the auto-evict will not be triggered for any node , since in this case LINSTOR assumes connection problems from the controller. This property can only be set for the controller, and only accepts a value between 0 and 100. The default value is 34. If you want to turn the auto-evict-feature off, simply set this property to 0. If you want to always trigger the auto-evict, regardless of how many satellites are unreachable, set it to 100.

  • DrbdOptions/AutoEvictAllowEviction is an additional property that can stop a node from being evicted. This can be useful for various cases, for example if you need to shut down a node for maintenance. You can set this property on the controller to change a global default, or on a single node to give it a different behavior. It accepts true and false as values and per default is set to true on the controller. You can use this property to turn the auto-evict feature off by setting it to false on the controller, although this might not work completely if you already set different values for individual nodes, since those values take precedence over the global default.

After the linstor-controller loses the connection to a satellite, aside from trying to reconnect, it starts a timer for that satellite. As soon as that timer exceeds DrbdOptions/AutoEvictAfterTime and all of the DRBD-connections to the DRBD-resources on that satellite are broken, the controller will check whether or not DrbdOptions/AutoEvictMaxDisconnectedNodes has been met. If it hasn’t, and DrbdOptions/AutoEvictAllowEviction is true for the node in question, the satellite will be marked as EVICTED. At the same time, the controller will check for every DRBD-resource whether the number of resources is still above DrbdOptions/AutoEvictMinReplicaCount. If it is, the resource in question will be marked as DELETED. If it isn’t, an auto-place with the settings from the corresponding resource-group will be started. Should the auto-place fail, the controller will try again later when changes that might allow a different result, such as adding a new node, have happened. Resources where an auto-place is necessary will only be marked as DELETED if the corresponding auto-place was successful.

The evicted satellite itself will not be able to reestablish connection with the controller. Even if the node is up and running, a manual reconnect will fail. It is also not possible to delete the satellite, even if it is working as it should be. The satellite can, however, be restored. This will remove the EVICTED-flag from the satellite and allow you to use it again. Previously configured network interfaces, storage pools, properties and similar entities as well as non-DRBD-related resources and resources that could not be autoplaced somewhere else will still be on the satellite. To restore a satellite, use

# linstor node restore [nodename]
  Should you want to instead throw everything that once was on that node,
including the node itself, away, you need to use the `node lost` command
instead.  [[s-linstor-auto-diskful]]
==== Auto-Diskful and Related Options
  You can set the LINSTOR `auto-diskful` and `auto-diskful-allow-cleanup`
properties for various LINSTOR objects, for example, a resource definition,
to have LINSTOR help automatically make a _Diskless_ node _Diskful_ and
perform appropriate cleanup actions afterwards.  This is useful when a
_Diskless_ node has been in a _Primary_ state for a DRBD resource for more
than a specified number of minutes. This could happen in cases where you
integrate LINSTOR managed storage with other orchestrating and scheduling
platforms, such as OpenStack, OpenNebula, and others. On some platforms that
you integrate LINSTOR with, you might not have a way to influence where in
your cluster a storage volume will be used.  The auto-diskful options give
you a way to use LINSTOR to sensibly delegate the roles of your storage
nodes in response to an integrated platform's actions that are beyond your
control.  [[s-linstor-auto-diskful-setting]]
===== Setting the Auto-Diskful Option

By setting the DrbdOptions/auto-diskful option on a LINSTOR resource definition , you are configuring the LINSTOR controller to make a Diskless DRBD resource Diskful if the resource has been in a DRBD Primary state for more than the specified number of minutes. After the specified number of minutes, LINSTOR will automatically use the resource toggle-disk command to toggle the resource state on the Diskless node, for the given resource.

To set this property, for example, on a LINSTOR resource definition named myres with a threshold of five minutes, enter the command:

# linstor resource-definition set-property myres DrbdOptions/auto-diskful 5
  [[s-linstor-auto-diskful-setting-on-rg-or-controller]]
===== Setting the Auto-Diskful Option on a Resource Group or Controller
  You can also set the `DrbdOptions/auto-diskful` option on LINSTOR
`controller` or `resource-group` objects. By setting the option at the
controller level, the option will affect all LINSTOR resource definitions in
your LINSTOR cluster that do not have this option set, either on the
resource definition itself, or else on the resource group that you might
have created the resource from.  Setting the option on a LINSTOR resource
group will affect all resource definitions that are spawned from the group,
unless a resource definition has the option set on it.  The order of
priority, from highest to lowest, for the effect of setting the
`auto-diskful` option is: - Resource definition - Resource group -
Controller [[s-linstor-auto-diskful-unsetting]]
===== Unsetting the Auto-Diskful Option

To unset the LINSTOR auto-diskful option, enter:

# linstor <controller|resource-definition|resource-group> set-property DrbdOptions/auto-diskful
  [[s-linstor-auto-diskful-allow-cleanup-setting]]
===== Setting the Auto-Diskful-Allow-Cleanup Option
  A companion option to the LINSTOR `auto-diskful` option is the
`DrbdOptions/auto-diskful-allow-cleanup` option.  You can set this option on
the following LINSTOR objects: node, resource, resource definition, or
resource group. The default value for this option is `True`, but the option
has no effect unless the `auto-diskful` option has also been set.  After
LINSTOR has toggled a resource to _Diskful_, because the threshold number of
minutes has passed where a _Diskless_ node was in the _Primary_ role for a
resource, and after DRBD has synchronized the data to this previously
_Diskless_ and now _Primary_ node, LINSTOR will remove the resource from any
_Secondary_ nodes when that action is necessary to fulfill a replica count
constraint that the resource might have. This could be the case, for
example, if you have specified a number of replicas for a resource by using
the `--auto-place` option.  [s-linstor-qos]]
=== QoS设置
  LINSTOR implements QoS for managed resources by using sysfs properties
that correspond to kernel variables related to block I/O operations. These
sysfs properties can be limits on either bandwidth (bytes per second), or
IOPS, or both.  The sysfs files and their corresponding LINSTOR properties
are as follows: [cols="3,2"]
sysfs (/sys/fs/) LINSTOR Property

cgroup/blkio/blkio.throttle.read_bps_device

sys/fs/blkio_throttle_read

cgroup/blkio/blkio.throttle.write_bps_device

sys/fs/blkio_throttle_write

cgroup/blkio/blkio.throttle.read_iops_device

sys/fs/blkio_throttle_read_iops

cgroup/blkio/blkio.throttle.write_iops_device

sys/fs/blkio_throttle_write_iops

2.15.5. Setting QoS Using LINSTOR sysfs Properties

These LINSTOR properties may be set using the set-property command and may be set on the following objects: volume, storage pool, resource, controller, or node. You can also set these QoS properties on resource groups, volume groups, resource definitions, or volume definitions. When you set a QoS property on a group or definition, resources created from the group or definition will inherit the QoS settings.

Settings made to a group or definition will affect both existing and new resources created from the group or definition.

The following example shows creating a resource group, then creating a volume group, then applying QoS settings to the volume group, and then spawning resources from the resource group. A verification command will show that the spawned resources inherit the QoS settings. The example uses an assumed previously created LINSTOR storage pool named pool1. You will need to replace this name with a storage pool name that exists in your environment.

# linstor resource-group create qos_limited --storage-pool pool1 --place-count 3
# linstor volume-group create qos_limited
# linstor volume-group set-property qos_limited 0 sys/fs/blkio_throttle_write 1048576
# linstor resource-group spawn-resources qos_limited qos_limited_res 200M

To verify that the spawned resources inherited the QoS setting, you can show the contents of the corresponding sysfs file, on a node that contributes storage to the storage pool.

# cat /sys/fs/cgroup/blkio/blkio.throttle.write_bps_device
252:4 1048576
As the QoS properties are inherited and not copied, you will not see the property listed in any “child” objects that have been spawned from the “parent” group or definition.

2.15.6. QoS Settings for a LINSTOR Volume Having Multiple DRBD Devices

A single LINSTOR volume can be composed of multiple DRBD devices. For example, DRBD with external metadata will have three backing devices: a data (storage) device, a metadata device, and the composite DRBD device (volume) provided to LINSTOR. If the data and metadata devices correspond to different backing disks, then if you set a sysfs property for such a LINSTOR volume, only the local data (storage) backing device will receive the property value in the corresponding /sys/fs/cgroup/blkio/ file. Neither the device backing DRBD’s metadata, nor the composite backing device provided to LINSTOR would receive the value. However, when DRBD’s data and its metadata share the same backing disk, QoS settings will affect the performance of both data and metadata operations.

2.15.7. QoS Settings for NVMe

In case a LINSTOR resource definition has an nvme-target as well as an nvme-initiator resource, both data (storage) backing devices of each node will receive the sysfs property value. In case of the target, the data backing device will be the volume of either LVM or ZFS, whereas in case of the initiator, the data backing device will be the connected nvme-device, regardless of which other LINSTOR layers, such as LUKS, NVMe, DRBD, and others (see Using LINSTOR Without DRBD), are above that.

2.16. Getting Help

2.16.1. From the Command Line

在命令行中列出可用命令的一种快速方法是键入 linstor

Further information about subcommands (e.g., list-nodes) can be retrieved in two ways:

# linstor node list -h
# linstor help node list

当LINSTOR以交互模式(LINSTOR interactive)执行时,使用 help 子命令尤其有用。

LINSTOR最有用的特性之一是它丰富的tab-completion,它基本上可以用来完成LINSTOR所知道的每个对象(例如,节点名、IP地址、资源名等等)。在下面的示例中,我们将展示一些可能的完成及其结果:

# linstor node create alpha 1<tab> # completes the IP address if hostname can be resolved
# linstor resource create b<tab> c<tab> # linstor assign-resource backups charlie

如果制表符完成不正常,请尝试获取相应的文件:

# source /etc/bash_completion.d/linstor # or
# source /usr/share/bash_completion/completions/linstor

For zsh shell users, the linstor-client command can generate a zsh compilation file, that has basic support for command and argument completion.

# linstor gen-zsh-completer > /usr/share/zsh/functions/Completion/Linux/_linstor

2.16.2. SOS-Report

If something goes wrong and you need help finding the cause of the issue, you can use

# linstor sos-report create

The command above will create a new sos-report in /var/log/linstor/controller/ on the controller node. Alternatively you can use

# linstor sos-report download

which will create a new sos-report and additionally downloads that report to the local machine into your current working directory.

This sos-report contains logs and useful debug-information from several sources (Linstor-logs, dmesg, versions of external tools used by LINSTOR, ip a, database dump and many more). These information are stored for each node in plain text in the resulting .tar.gz file.

2.16.3. From the Community

如需社区帮助,请订阅我们的邮件列表: https://lists.linbit.com/listinfo/drbd-user

2.16.4. Github

要提交bug或功能请求,请查看我们的GitHub页面 https://GitHub.com/linbit

2.16.5. Paid Support and Development

Alternatively, if you need to purchase remote installation services, 24/7 support, access to certified repositories, or feature development, please contact us: +1-877-454-6248 (1-877-4LINBIT) , International: +43-1-8178292-0 | [email protected]

3. Kubernetes的LINSTOR卷

This chapter describes the usage of LINSTOR in Kubernetes (K8s) as managed by the operator and with volumes provisioned using the LINSTOR CSI plug-in.

This Chapter goes into great detail regarding all the install time options and various configurations possible with LINSTOR and Kubernetes. For those more interested in a “quick-start” for testing, or those looking for some examples for reference. We have some complete Helm Install Examples of a few common uses near the end of the chapter.

3.1. Kubernetes Introduction

Kubernetes is a container orchestrator. Kubernetes defines the behavior of containers and related services, using declarative specifications. In this guide, we will focus on using kubectl to manipulate YAML files that define the specifications of Kubernetes objects.

3.2. 在Kubernetes上部署LINSTOR

LINBIT provides a LINSTOR Operator to commercial support customers. The Operator eases deployment of LINSTOR on Kubernetes by installing DRBD, managing Satellite and Controller pods, and other related functions.

LINBIT’s container image repository (https://drbd.io), used by LINSTOR Operator, is only available to LINBIT customers or through LINBIT customer trial accounts. Contact LINBIT for information on pricing or to begin a trial. Alternatively, you may use LINSTOR SDS’ upstream project named Piraeus, without being a LINBIT customer.

LINSTOR Operator v2 is the recommended way of deploying LINBIT SDS for Kubernetes on new clusters. Users of existing Operator v1 deployments should continue to use their Helm deployments, skipping to the Operator v1 instructions.

3.2.1. Deploying LINSTOR Operator v2

LINSTOR Operator v2 is deployed using kustomize tool, integrated with kubectl.

Prerequisites for LINSTOR Operator v2

To deploy LINSTOR Operator v2, you need to meet the following prerequisites:

Creating the Operator

To deploy the Operator, create a kustomization.yaml file. This will declare your pull secret for drbd.io and allow you to pull in the Operator deployment. The Operator will be deployed in a new namespace linbit-sds. Make sure to replace MY_LINBIT_USER and MY_LINBIT_PASSWORD with your own credentials.

Listing 1. kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: linbit-sds
resources:
  - https://github.com/LINBIT/linstor-operator-builder//deploy/default?ref=v2.0.1
generatorOptions:
  disableNameSuffixHash: true
secretGenerator:
  - name: drbdio-pull-secret
    type: kubernetes.io/dockerconfigjson
    literals:
      - .dockerconfigjson={"auths":{"drbd.io":{"username":"MY_LINBIT_USER","password":"MY_LINBIT_PASSWORD"}}}

Then, apply the kustomization.yaml file, by using kubectl command, and wait for the Operator to start:

$ kubectl apply -k .
namespace/linbit-sds created
...
$ kubectl -n linbit-sds  wait pod --for=condition=Ready --all
pod/linstor-operator-controller-manager-6d9847d857-zc985 condition met

The Operator is now ready to deploy LINBIT SDS for Kubernetes.

Deploying LINBIT SDS for Kubernetes

Deploying LINBIT SDS for Kubernetes with the Operator is a simple as creating a new LinstorCluster resource and waiting for the Operator to complete the setup:

$ kubectl create -f - <<EOF
apiVersion: piraeus.io/v1
kind: LinstorCluster
metadata:
  name: linstorcluster
spec: {}
EOF
$ kubectl wait pod --for=condition=Ready -n linbit-sds --timeout=3m --all
pod/ha-controller-4tgcg condition met
pod/k8s-1-26-10.test condition met
pod/linstor-controller-76459dc6b6-tst8p condition met
pod/linstor-csi-controller-75dfdc967d-dwdx6 condition met
pod/linstor-csi-node-9gcwj condition met
pod/linstor-operator-controller-manager-6d9847d857-zc985 condition met
Configuring Storage

By default, LINBIT SDS for Kubernetes does not configure any storage. To add storage, you can configure a LinstorSatelliteConfiguration, which the Operator uses to configure one or more satellites.

The following example creates a simple FILE_THIN pool and it does not require any additional set up on the host:

$ kubectl apply -f - <<EOF
apiVersion: piraeus.io/v1
kind: LinstorSatelliteConfiguration
metadata:
  name: storage-pool
spec:
  storagePools:
    - name: pool1
      fileThinPool:
        directory: /var/lib/linbit-sds/pool1
EOF

Other types of storage pools can be configured as well. Please check the examples upstream.

Next Steps

After deploying LINBIT SDS for Kubernetes, you can continue with the guide for creating basic volumes, or refer to the available tutorials and how-to guides upstream.

3.2.2. Deploying LINSTOR Operator v1

If you plan to deploy LINSTOR Operator on a new cluster, you should use Operator v2.

The Operator v1 is installed using a Helm v3 chart as follows:

  • Create a Kubernetes secret containing your my.linbit.com credentials:

    kubectl create secret docker-registry drbdiocred --docker-server=drbd.io \
      --docker-username=<YOUR_LOGIN> --docker-email=<YOUR_EMAIL> --docker-password=<YOUR_PASSWORD>

    默认情况下,此密码的名称必须与Helm值中指定的名称 drbdiocred 匹配。

  • Configure the LINSTOR database back end. By default, the chart configures etcd as database back end. The Operator can also configure LINSTOR to use Kubernetes as datastore directly. If you go the etcd route, you should configure persistent storage for it:

    • 使用具有默认 StorageClass 的现有存储资源调配器。

    • Use hostPath volumes

    • Disable persistence, for basic testing only. This can be done by adding --set etcd.persistentVolume.enabled=false to the helm install command below.

  • Read the storage guide and configure a basic storage setup for LINSTOR

  • Read the section on securing the deployment and configure as needed.

  • 在最后一步中,使用带有 helm install 命令的 --set 选择适当的内核模块注入。

    • Choose the injector according to the distribution you are using. Select the latest version from one of drbd9-rhel7, drbd9-rhel8,…​ from http://drbd.io/ as appropriate. The drbd9-rhel8 image should also be used for RHCOS (OpenShift). For the SUSE CaaS Platform use the SLES injector that matches the base system of the CaaS Platform you are using (e.g., drbd9-sles15sp1). For example:

      operator.satelliteSet.kernelModuleInjectionImage=drbd.io/drbd9-rhel8:v9.1.8
    • Only inject modules that are already present on the host machine. If a module is not found, it will be skipped.

      operator.satelliteSet.kernelModuleInjectionMode=DepsOnly
    • Disable kernel module injection if you are installing DRBD by other means. Deprecated by DepsOnly

      operator.satelliteSet.kernelModuleInjectionMode=None
  • 最后创建一个名为 linstor-op 的Helm部署,它将设置所有内容。

    helm repo add linstor https://charts.linstor.io
    helm install linstor-op linstor/linstor

    Further deployment customization is discussed in the advanced deployment section

Kubernetes Back End for LINSTOR

The Controller can use the Kubernetes API directly to persist its cluster state. To enable this back end, use the following override file during the chart installation:

Listing 2. k8s-backend.yaml
etcd:
  enabled: false
operator:
  controller:
    dbConnectionURL: k8s
It is NOT possible to migrate from an existing cluster with etcd back end to the Kubernetes back end.
Creating Persistent Storage Volumes

You can use the pv-hostpath Helm templates to create hostPath persistent volumes. Create as many PVs as needed to satisfy your configured etcd replicas (default 1).

创建 hostPath 持久卷,在 nodes= 选项中相应地替换为集群节点名称:

helm repo add linstor https://charts.linstor.io
helm install linstor-etcd linstor/pv-hostpath

By default, a PV is created on every control-plane node. You can manually select the storage nodes by passing --set "nodes={<NODE0>,<NODE1>,<NODE2>}" to the install command.

The correct value to reference the node is the value of the kubernetes.io/hostname label. You can list the value for all nodes by running kubectl get nodes -o custom-columns="Name:{.metadata.name},NodeName:{.metadata.labels['kubernetes\.io/hostname']}"
Using an Existing Database

LINSTOR can connect to an existing PostgreSQL, MariaDB or etcd database. For instance, for a PostgreSQL instance with the following configuration:

POSTGRES_DB: postgresdb
POSTGRES_USER: postgresadmin
POSTGRES_PASSWORD: admin123

The Helm chart can be configured to use this database rather than deploying an etcd cluster, by adding the following to the Helm install command:

--set etcd.enabled=false --set "operator.controller.dbConnectionURL=jdbc:postgresql://postgres/postgresdb?user=postgresadmin&password=admin123"

3.2.3. Configuring Storage

The LINSTOR Operator can automate some basic storage set up for LINSTOR.

Configuring Storage Pool Creation

The LINSTOR Operator can be used to create LINSTOR storage pools. Creation is under control of the LinstorSatelliteSet resource:

$ kubectl get LinstorSatelliteSet.linstor.linbit.com linstor-op-ns -o yaml
kind: LinstorSatelliteSet
metadata:
..
spec:
  ..
  storagePools:
    lvmPools:
    - name: lvm-thick
      volumeGroup: drbdpool
    lvmThinPools:
    - name: lvm-thin
      thinVolume: thinpool
      volumeGroup: ""
    zfsPools:
    - name: my-linstor-zpool
      zPool: for-linstor
      thin: true
Creating Storage Pools at Installation Time

At installation time, by setting the value of operator.satelliteSet.storagePools when running the helm install command.

First create a file with the storage configuration like:

operator:
  satelliteSet:
    storagePools:
      lvmPools:
      - name: lvm-thick
        volumeGroup: drbdpool

This file can be passed to the Helm installation like this:

helm install -f <file> linstor-op linstor/linstor
Creating Storage Pools After Installation

On a cluster with the operator already configured (that is, after helm install), you can edit the LinstorSatelliteSet configuration like this:

$ kubectl edit LinstorSatelliteSet.linstor.linbit.com <satellitesetname>

The storage pool configuration can be updated like in the example above.

Preparing Physical Devices

By default, LINSTOR expects the referenced VolumeGroups, ThinPools and so on to be present. You can use the devicePaths: [] option to let LINSTOR automatically prepare devices for the pool. Eligible for automatic configuration are block devices that:

  • Are a root device (no partition)

  • do not contain partition information

  • have more than 1 GiB

To enable automatic configuration of devices, set the devicePaths key on storagePools entries:

  storagePools:
    lvmPools:
    - name: lvm-thick
      volumeGroup: drbdpool
      devicePaths:
      - /dev/vdb
    lvmThinPools:
    - name: lvm-thin
      thinVolume: thinpool
      volumeGroup: linstor_thinpool
      devicePaths:
      - /dev/vdc
      - /dev/vdd

Currently, this method supports creation of LVM and LVMTHIN storage pools.

Configuring LVM Storage Pools

The available keys for lvmPools entries are:

  • name name of the LINSTOR storage pool. [Required]

  • volumeGroup name of the VG to create. [Required]

  • devicePaths devices to configure for this pool. Must be empty and >= 1GiB to be recognized. [Optional]

  • raidLevel LVM raid level. [Optional]

  • vdo Enable [VDO] (requires VDO tools in the satellite). [Optional]

  • vdoLogicalSizeKib Size of the created VG (expected to be bigger than the backing devices by using VDO). [Optional]

  • vdoSlabSizeKib Slab size for VDO. [Optional]

Configuring LVM Thin Pools
  • name name of the LINSTOR storage pool. [Required]

  • volumeGroup VG to use for the thin pool. If you want to use devicePaths, you must set this to "". This is required because LINSTOR does not allow configuration of the VG name when preparing devices. thinVolume name of the thin pool. [Required]

  • devicePaths devices to configure for this pool. Must be empty and >= 1GiB to be recognized. [Optional]

  • raidLevel LVM raid level. [Optional]

The volume group created by LINSTOR for LVM thin pools will always follow the scheme “linstor_$THINPOOL”.
Configuring ZFS Storage Pools
  • name name of the LINSTOR storage pool. [Required]

  • zPool name of the zpool to use. Must already be present on all machines. [Required]

  • thin true to use thin provisioning, false otherwise. [Required]

Automatic Storage Type Provisioning (DEPRECATED)

ALL eligible devices will be prepared according to the value of operator.satelliteSet.automaticStorageType, unless they are already prepared using the storagePools section. Devices are added to a storage pool based on the device name (that is, all /dev/nvme1 devices will be part of the pool autopool-nvme1)

The possible values for operator.satelliteSet.automaticStorageType:

  • None no automatic set up (default)

  • LVM create a LVM (thick) storage pool

  • LVMTHIN create a LVM thin storage pool

  • ZFS create a ZFS based storage pool (UNTESTED)

3.2.4. Securing Deployment

This section describes the different options for enabling security features available when using this operator. The following guides assume the operator is installed using Helm

Secure Communication with an Existing etcd Instance

Secure communication to an etcd instance can be enabled by providing a CA certificate to the operator in form of a kubernetes secret. The secret has to contain the key ca.pem with the PEM encoded CA certificate as value.

The secret can then be passed to the controller by passing the following argument to helm install

--set operator.controller.dbCertSecret=<secret name>
Authentication with etcd Using Certificates

If you want to use TLS certificates to authenticate with an etcd database, you need to set the following option on Helm install:

--set operator.controller.dbUseClientCert=true

If this option is active, the secret specified in the above section must contain two additional keys:

  • client.cert PEM formatted certificate presented to etcd for authentication

  • client.key private key in PKCS8 format, matching the above client certificate.

Keys can be converted into PKCS8 format using openssl:

openssl pkcs8 -topk8 -nocrypt -in client-key.pem -out client-key.pkcs8
Configuring Secure Communication Between LINSTOR Components

The default communication between LINSTOR components is not secured by TLS. If this is needed for your setup, choose one of three methods:

Generating Keys and Certificates Using cert-manager

Requires cert-manager to be installed in your cluster.

Set the following options in your Helm override file:

linstorSslMethod: cert-manager
linstorHttpsMethod: cert-manager
Generate Keys and Certificates Using Helm

Set the following options in your Helm override file:

linstorSslMethod: helm
linstorHttpsMethod: helm
Generating Keys and Certificates Manually

Create a private key and self-signed certificate for your certificate authorities:

openssl req -new -newkey rsa:2048 -days 5000 -nodes -x509 -keyout ca.key \
  -out ca.crt -subj "/CN=linstor-system"
openssl req -new -newkey rsa:2048 -days 5000 -nodes -x509 -keyout client-ca.key \
  -out client-ca.crt -subj "/CN=linstor-client-ca"

Create private keys, two for the controller, one for all nodes and one for all clients:

openssl genrsa -out linstor-control.key 2048
openssl genrsa -out linstor-satellite.key 2048
openssl genrsa -out linstor-client.key 2048
openssl genrsa -out linstor-api.key 2048

Create trusted certificates for controller and nodes:

openssl req -new -sha256 -key linstor-control.key -subj "/CN=system:control" \
  -out linstor-control.csr
openssl req -new -sha256 -key linstor-satellite.key -subj "/CN=system:node" \
  -out linstor-satellite.csr
openssl req -new -sha256 -key linstor-client.key -subj "/CN=linstor-client" \
  -out linstor-client.csr
openssl req -new -sha256 -key linstor-api.key -subj "/CN=linstor-controller" \
  -out  linstor-api.csr
openssl x509 -req -in linstor-control.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out linstor-control.crt -days 5000 -sha256
openssl x509 -req -in linstor-satellite.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out linstor-satellite.crt -days 5000 -sha256
openssl x509 -req -in linstor-client.csr -CA client-ca.crt -CAkey client-ca.key \
  -CAcreateserial -out linstor-client.crt -days 5000 -sha256
openssl x509 -req -in linstor-api.csr -CA client-ca.crt -CAkey client-ca.key \
  -CAcreateserial -out linstor-api.crt -days 5000 -sha256 -extensions 'v3_req' \
  -extfile <(printf '%s\n' '[v3_req]' extendedKeyUsage=serverAuth \
  subjectAltName=DNS:linstor-op-cs.default.svc)
linstor-op-cs.default.svc in the last command needs to match create service name. With Helm, this is always <release-name>-cs.<namespace>.svc.

Create kubernetes secrets that can be passed to the controller and node pods:

kubectl create secret generic linstor-control --type=kubernetes.io/tls \
  --from-file=ca.crt=ca.crt --from-file=tls.crt=linstor-control.crt \
  --from-file=tls.key=linstor-control.key
kubectl create secret generic linstor-satellite --type=kubernetes.io/tls \
  --from-file=ca.crt=ca.crt --from-file=tls.crt=linstor-satellite.crt \
  --from-file=tls.key=linstor-satellite.key
kubectl create secret generic linstor-api --type=kubernetes.io/tls \
  --from-file=ca.crt=client-ca.crt --from-file=tls.crt=linstor-api.crt \
  --from-file=tls.key=linstor-api.key
kubectl create secret generic linstor-client --type=kubernetes.io/tls \
  --from-file=ca.crt=client-ca.crt --from-file=tls.crt=linstor-client.crt \
  --from-file=tls.key=linstor-client.key

Pass the names of the created secrets to helm install:

linstorHttpsControllerSecret: linstor-api
linstorHttpsClientSecret: linstor-client
operator:
  controller:
    sslSecret: linstor-control
  satelliteSet:
    sslSecret: linstor-satellite
Automatically Set the Passphrase for LINSTOR

LINSTOR needs to store confidential data to support encrypted information. This data is protected by a master passphrase. A passphrase is automatically generated on the first chart install.

If you want to use a custom passphrase, store it in a secret:

kubectl create secret generic linstor-pass --from-literal=MASTER_PASSPHRASE=<password>

On install, add the following arguments to the Helm command:

--set operator.controller.luksSecret=linstor-pass
Helm Install Examples

All the below examples use the following sp-values.yaml file. Feel free to adjust this for your uses and environment. See [Configuring storage pool creation] for further details.

operator:
  satelliteSet:
    storagePools:
      lvmThinPools:
      - name: lvm-thin
        thinVolume: thinpool
        volumeGroup: ""
        devicePaths:
        - /dev/sdb

Default install. Please note this does not setup any persistence for the backing etcd key-value store.

This is not suggested for any use outside of testing.
kubectl create secret docker-registry drbdiocred --docker-server=drbd.io \
  --docker-username=<YOUR_LOGIN> --docker-password=<YOUR_PASSWORD>
helm repo add linstor https://charts.linstor.io
helm install linstor-op linstor/linstor
LINBIT’s container image repository (http://drbd.io), used in the previous and upcoming kubectl create commands, is only available to LINBIT customers or through LINBIT customer trial accounts. Contact LINBIT for information on pricing or to begin a trial. Alternatively, you may use LINSTOR SDS’ upstream project named Piraeus, without being a LINBIT customer.

Install with LINSTOR storage-pools defined at install through sp-values.yaml, persistent hostPath volumes, 3 etcd replicas, and by compiling the DRBD kernel modules for the host kernels.

This should be adequate for most basic deployments. Please note that this deployment is not using the pre-compiled DRBD kernel modules just to make this command more portable. Using the pre-compiled binaries will make for a much faster install and deployment. Using the Compile option would not be suggested for use in a large Kubernetes clusters.

kubectl create secret docker-registry drbdiocred --docker-server=drbd.io \
  --docker-username=<YOUR_LOGIN> --docker-password=<YOUR_PASSWORD>
helm repo add linstor https://charts.linstor.io
helm install linstor-etcd linstor/pv-hostpath --set "nodes={<NODE0>,<NODE1>,<NODE2>}"
helm install -f sp-values.yaml linstor-op linstor/linstor --set etcd.replicas=3 \
  --set operator.satelliteSet.kernelModuleInjectionMode=Compile

Install with LINSTOR storage-pools defined at install through sp-values.yaml, use an already created PostgreSQL DB (preferably clustered), rather than etcd, and use already compiled kernel modules for DRBD.

The PostgreSQL database in this particular example is reachable through a service endpoint named postgres. PostgreSQL itself is configured with POSTGRES_DB=postgresdb, POSTGRES_USER=postgresadmin, and POSTGRES_PASSWORD=admin123

kubectl create secret docker-registry drbdiocred --docker-server=drbd.io \
  --docker-username=<YOUR_LOGIN> --docker-email=<YOUR_EMAIL> --docker-password=<YOUR_PASSWORD>
helm repo add linstor https://charts.linstor.io
helm install -f sp-values.yaml linstor-op linstor/linstor --set etcd.enabled=false \
  --set "operator.controller.dbConnectionURL=jdbc:postgresql://postgres/postgresdb?user=postgresadmin&password=admin123"
Terminating Helm Deployment

To protect the storage infrastructure of the cluster from accidentally deleting vital components, it is necessary to perform some manual steps before deleting a Helm deployment.

  1. Delete all volume claims managed by LINSTOR components. You can use the following command to get a list of volume claims managed by LINSTOR. After checking that none of the listed volumes still hold needed data, you can delete them using the generated kubectl delete command.

    $ kubectl get pvc --all-namespaces -o=jsonpath='{range .items[?(@.metadata.annotations.volume\.beta\.kubernetes\.io/storage-provisioner=="linstor.csi.linbit.com")]}kubectl delete pvc --namespace {.metadata.namespace} {.metadata.name}{"\n"}{end}'
    kubectl delete pvc --namespace default data-mysql-0
    kubectl delete pvc --namespace default data-mysql-1
    kubectl delete pvc --namespace default data-mysql-2
    These volumes, once deleted, cannot be recovered.
  2. Delete the LINSTOR controller and satellite resources.

    Deployment of LINSTOR satellite and controller is controlled by the LinstorSatelliteSet and LinstorController resources. You can delete the resources associated with your deployment using kubectl

    kubectl delete linstorcontroller <helm-deploy-name>-cs
    kubectl delete linstorsatelliteset <helm-deploy-name>-ns

    After a short wait, the controller and satellite pods should terminate. If they continue to run, you can check the above resources for errors (they are only removed after all associated pods have terminated).

  3. Delete the Helm deployment.

    If you removed all PVCs and all LINSTOR pods have terminated, you can uninstall the Helm deployment

    helm uninstall linstor-op
    Due to the Helm’s current policy, the Custom Resource Definitions named LinstorController and LinstorSatelliteSet will not be deleted by the command. More information regarding Helm’s current position on CRDs can be found here.

3.2.5. Advanced Deployment Options

The Helm charts provide a set of further customization options for advanced use cases.

LINBIT’s container image repository (http://drbd.io), used in the Helm chart below, is only available to LINBIT customers or through LINBIT customer trial accounts. Contact LINBIT for information on pricing or to begin a trial. Alternatively, you may use LINSTOR SDS’ upstream project named Piraeus, without being a LINBIT customer.
global:
  imagePullPolicy: IfNotPresent # empty pull policy means k8s default is used ("always" if tag == ":latest", "ifnotpresent" else) (1)
  setSecurityContext: true # Force non-privileged containers to run as non-root users
# Dependency charts
etcd:
  enabled: true
  persistentVolume:
    enabled: true
    storage: 1Gi
  replicas: 1 # How many instances of etcd will be added to the initial cluster. (2)
  resources: {} # resource requirements for etcd containers (3)
  image:
    repository: gcr.io/etcd-development/etcd
    tag: v3.4.15
stork:
  enabled: false
  storkImage: docker.io/openstorage/stork:2.8.2
  schedulerImage: k8s.gcr.io/kube-scheduler-amd64
  schedulerTag: ""
  replicas: 1 (2)
  storkResources: {} # resources requirements for the stork plug-in containers (3)
  schedulerResources: {} # resource requirements for the kube-scheduler containers (3)
  podsecuritycontext: {}
csi:
  enabled: true
  pluginImage: "drbd.io/linstor-csi:v0.20.0"
  csiAttacherImage: k8s.gcr.io/sig-storage/csi-attacher:v4.0.0
  csiLivenessProbeImage: k8s.gcr.io/sig-storage/livenessprobe:v2.7.0
  csiNodeDriverRegistrarImage: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.1
  csiProvisionerImage: k8s.gcr.io/sig-storage/csi-provisioner:v3.2.1
  csiSnapshotterImage: k8s.gcr.io/sig-storage/csi-snapshotter:v6.0.1
  csiResizerImage: k8s.gcr.io/sig-storage/csi-resizer:v1.6.0
  csiAttacherWorkerThreads: 10 (9)
  csiProvisionerWorkerThreads: 10 (9)
  csiSnapshotterWorkerThreads: 10 (9)
  csiResizerWorkerThreads: 10 (9)
  controllerReplicas: 1 (2)
  nodeAffinity: {} (4)
  nodeTolerations: [] (4)
  controllerAffinity: {} (4)
  controllerTolerations: [] (4)
  enableTopology: true
  resources: {} (3)
  customLabels: {}
  customAnnotations: {}
  kubeletPath: /var/lib/kubelet (7)
  controllerSidecars: []
  controllerExtraVolumes: []
  nodeSidecars: []
  nodeExtraVolumes: []
priorityClassName: ""
drbdRepoCred: drbdiocred
linstorSslMethod: "manual" # <- If set to 'helm' or 'cert-manager' the certificates will be generated automatically
linstorHttpsMethod: "manual" # <- If set to 'helm' or 'cert-manager' the certificates will be generated automatically
linstorHttpsControllerSecret: "" # <- name of secret containing linstor server certificates+key. See docs/security.md
linstorHttpsClientSecret: "" # <- name of secret containing linstor client certificates+key. See docs/security.md
controllerEndpoint: "" # <- override to the generated controller endpoint. use if controller is not deployed via operator
psp:
  privilegedRole: ""
  unprivilegedRole: ""
operator:
  replicas: 1 # <- number of replicas for the operator deployment (2)
  image: "drbd.io/linstor-operator:v1.10.0"
  affinity: {} (4)
  tolerations: [] (4)
  resources: {} (3)
  customLabels: {}
  customAnnotations: {}
  podsecuritycontext: {}
  args:
    createBackups: true
    createMonitoring: true
  sidecars: []
  extraVolumes: []
  controller:
    enabled: true
    controllerImage: "drbd.io/linstor-controller:v1.20.0"
    dbConnectionURL: ""
    luksSecret: ""
    dbCertSecret: ""
    dbUseClientCert: false
    sslSecret: ""
    affinity: {} (4)
    httpBindAddress: ""
    httpsBindAddress: ""
    tolerations: (4)
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
    resources: {} (3)
    replicas: 1 (2)
    additionalEnv: [] (5)
    additionalProperties: {} (6)
    sidecars: []
    extraVolumes: []
    customLabels: {}
    customAnnotations: {}
  satelliteSet:
    enabled: true
    satelliteImage: "drbd.io/linstor-satellite:v1.20.0"
    storagePools: {}
    sslSecret: ""
    automaticStorageType: None
    affinity: {} (4)
    tolerations: [] (4)
    resources: {} (3)
    monitoringImage: "drbd.io/drbd-reactor:v0.9.0"
    monitoringBindAddress: ""
    mountDrbdResourceDirectoriesFromHost: false (10)
    kernelModuleInjectionImage: "drbd.io/drbd9-rhel7:v9.1.11"
    kernelModuleInjectionMode: ShippedModules
    kernelModuleInjectionAdditionalSourceDirectory: "" (8)
    kernelModuleInjectionResources: {} (3)
    kernelModuleInjectionExtraVolumeMounts: []
    additionalEnv: [] (5)
    sidecars: []
    extraVolumes: []
    customLabels: {}
    customAnnotations: {}
haController:
  enabled: false
  image: drbd.io/linstor-k8s-ha-controller:v0.3.0
  affinity: {} (4)
  tolerations: [] (4)
  resources: {} (3)
  replicas: 1 (2)
  customLabels: {}
  customAnnotations: {}
1 Sets the pull policy for all images.
2 Controls the number of replicas for each component.
3 Set container resource requests and limits. See the kubernetes docs. Most containers need a minimal amount of resources, except for:
  • etcd.resources See the etcd docs

  • operator.controller.resources Around 700MiB memory is required

  • operater.satelliteSet.resources Around 700MiB memory is required operator.satelliteSet.kernelModuleInjectionResources If kernel modules are compiled, 1GiB of memory is required.

4 Affinity and toleration determine where pods are scheduled on the cluster. See the kubernetes docs on affinity and toleration. This may be especially important for the operator.satelliteSet and csi.node* values. To schedule a pod using a LINSTOR persistent volume, the node requires a running LINSTOR satellite and LINSTOR CSI pod.
5 Sets additional environments variables to pass to the LINSTOR Controller and Satellites. Uses the same format as the env value of a container
6 Sets additional properties on the LINSTOR Controller. Expects a simple mapping of <property-key>: <value>.
7 Kubelet expects every CSI plug-in to mount volumes under a specific subdirectory of its own state directory. By default, this state directory is /var/lib/kubelet. Some Kubernetes distributions use a different directory:
  • microk8s: /var/snap/microk8s/common/var/lib/kubelet

8 Directory on the host that is required for building kernel modules. Only needed if using the Compile injection method. Defaults to /usr/src, which is where the actual kernel sources are stored on most distributions. Use "none" to not mount any additional directories.
9 Set the number of worker threads used by the CSI driver. Higher values put more load on the LINSTOR Controller, which may lead to instability when creating many volumes at once.
10 If set to true, the satellite containers will have the following files and directories mounted from the host OS:
  • /etc/drbd/drbd.conf (file)

  • /etc/drbd.d (directory)

  • /var/lib/drbd (directory)

  • /var/lib/linstor.d (directory)

    All files and directories must already exist on the host.

High-Availability Deployment

To create a high-availability deployment of all components, consult the upstream guide The default values are chosen so that scaling the components to multiple replicas ensures that the replicas are placed on different nodes. This ensures that a single node failures will not interrupt the service.

3.2.6. Monitoring with Prometheus

You can use Prometheus to monitor LINSTOR components. The operator will set up monitoring containers along the existing components and make them available as a Service.

If you use the Prometheus Operator, the LINSTOR Operator will also set up the ServiceMonitor instances. The metrics will automatically be collected by the Prometheus instance associated to the operator, assuming watching the Piraeus namespace is enabled.

To disable exporting of metrics, set operator.satelliteSet.monitoringImage to an empty value.

LINSTOR Controller Monitoring

The LINSTOR Controller exports cluster-wide metrics. Metrics are exported on the existing controller service, using the path /metrics.

DRBD Resource Monitoring

All satellites are bundled with a secondary container that uses drbd-reactor to export metrics directly from DRBD. The metrics are available on port 9942, for convenience a headless service named <linstorsatelliteset-name>-monitoring is provided.

If you want to disable the monitoring container, set monitoringImage to "" in your LinstorSatelliteSet resource.

3.2.7. Deploying with an External LINSTOR Controller

The operator can configure the satellites and CSI plug-in to use an existing LINSTOR setup. This can be useful in cases where the storage infrastructure is separate from the Kubernetes cluster. Volumes can be provisioned in diskless mode on the Kubernetes nodes while the storage nodes will provide the backing disk storage.

To skip the creation of a LINSTOR Controller deployment and configure the other components to use your existing LINSTOR Controller, use the following options when running helm install:

  • operator.controller.enabled=false This disables creation of the LinstorController resource

  • operator.etcd.enabled=false Since no LINSTOR Controller will run on Kubernetes, no database is required.

  • controllerEndpoint=<url-of-linstor-controller> The HTTP endpoint of the existing LINSTOR Controller. For example: http://linstor.storage.cluster:3370/

After all pods are ready, you should see the Kubernetes cluster nodes as satellites in your LINSTOR setup.

Your kubernetes nodes must be reachable using their IP by the controller and storage nodes.

Create a storage class referencing an existing storage pool on your storage nodes.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-on-k8s
provisioner: linstor.csi.linbit.com
parameters:
  autoPlace: "3"
  storagePool: existing-storage-pool
  resourceGroup: linstor-on-k8s

You can provision new volumes by creating PVCs using your storage class. The volumes will first be placed only on nodes with the given storage pool, that is, your storage infrastructure. Once you want to use the volume in a pod, LINSTOR CSI will create a diskless resource on the Kubernetes node and attach over the network to the diskful resource.

3.2.8. 使用Piraeus Operator部署

在Kubernetes中由社区支持的LINSTOR部署版本称为Piraeus。Piraeus项目提供了一个 an operator 进行部署。

3.3. 在Kubernetes中与LINSTOR互动

Controller pod包括LINSTOR客户端,使得直接与LINSTOR交互变得容易。例如:

kubectl exec deployment/linstor-op-cs-controller -- linstor storage-pool list

For a convenient shortcut to the above command, download kubectl-linstor and install it alongside kubectl. Then you can use kubectl linstor to get access to the complete LINSTOR CLI.

$ kubectl linstor node list
╭────────────────────────────────────────────────────────────────────────────────────╮
┊ Node                           ┊ NodeType   ┊ Addresses                   ┊ State  ┊
╞════════════════════════════════════════════════════════════════════════════════════╡
┊ kube-node-01.test              ┊ SATELLITE  ┊ 10.43.224.26:3366 (PLAIN)   ┊ Online ┊
┊ kube-node-02.test              ┊ SATELLITE  ┊ 10.43.224.27:3366 (PLAIN)   ┊ Online ┊
┊ kube-node-03.test              ┊ SATELLITE  ┊ 10.43.224.28:3366 (PLAIN)   ┊ Online ┊
┊ linstor-op-cs-controller-[...] ┊ CONTROLLER ┊ 172.24.116.114:3366 (PLAIN) ┊ Online ┊
╰────────────────────────────────────────────────────────────────────────────────────╯

It also expands references to PVCs to the matching LINSTOR resource

$ kubectl linstor resource list -r pvc:my-namespace/demo-pvc-1 --all
pvc:my-namespace/demo-pvc-1 -> pvc-2f982fb4-bc05-4ee5-b15b-688b696c8526
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node              ┊ Port ┊ Usage  ┊ Conns ┊    State   ┊ CreatedOn           ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════╡
┊ pvc-[...]    ┊ kube-node-01.test ┊ 7000 ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2021-02-05 09:16:09 ┊
┊ pvc-[...]    ┊ kube-node-02.test ┊ 7000 ┊ Unused ┊ Ok    ┊ TieBreaker ┊ 2021-02-05 09:16:08 ┊
┊ pvc-[...]    ┊ kube-node-03.test ┊ 7000 ┊ InUse  ┊ Ok    ┊   UpToDate ┊ 2021-02-05 09:16:09 ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────╯

It also expands references of the form pod:[<namespace>/]<podname> into a list resources in use by the pod.

This should only be necessary for investigating problems and accessing advanced functionality. Regular operation such as creating volumes should be achieved through the Kubernetes integration.

3.4. 基本配置和部署

一旦所有linstor-csi Pods 都启动并运行,我们就可以使用通常的Kubernetes工作流创建卷。

Configuring the behavior and properties of LINSTOR volumes deployed through Kubernetes is accomplished using StorageClasses.

the “resourceGroup” parameter is mandatory. Usually you want it to be unique and the same as the storage class name.

Here below is the simplest practical StorageClass that can be used to deploy volumes:

Listing 3. linstor-basic-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  # The name used to identify this StorageClass.
  name: linstor-basic-storage-class
  # The name used to match this StorageClass with a provisioner.
  # linstor.csi.linbit.com is the name that the LINSTOR CSI plug-in uses to identify itself
provisioner: linstor.csi.linbit.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  # LINSTOR will provision volumes from the drbdpool storage pool configured
  # On the satellite nodes in the LINSTOR cluster specified in the plug-in's deployment
  storagePool: "lvm-thin"
  resourceGroup: "linstor-basic-storage-class"
  # Setting a fstype is required for "fsGroup" permissions to work correctly.
  # Currently supported: xfs/ext4
  csi.storage.k8s.io/fstype: xfs
The storagePool value, lvm-thin in the example YAML configuration file above, must match an available LINSTOR StoragePool. You can list storage pool information using the linstor storage-pool list command, executed within the running linstor-op-cs-controller pod.

我们可以使用以下命令创建 StorageClasses

kubectl create -f linstor-basic-sc.yaml

现在,我们的存储类已经创建,我们现在可以创建一个 PersistentVolumeClaim ,它可以用来提供Kubernetes和LINSTOR都知道的卷:

Listing 4. my-first-linstor-volume-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: my-first-linstor-volume
spec:
  storageClassName: linstor-basic-storage-class
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi

我们可以使用以下命令创建 PersistentVolumeClaim

kubectl create -f my-first-linstor-volume-pvc.yaml

This will create a PersistentVolumeClaim, but no volume will be created just yet. The storage class we used specified volumeBindingMode: WaitForFirstConsumer, which means that the volume is only created once a workload starts using it. This ensures that the volume is placed on the same node as the workload.

For our example, we create a simple Pod, which mounts or volume by referencing the PersistentVolumeClaim. .my-first-linstor-volume-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: fedora
  namespace: default
spec:
  containers:
  - name: fedora
    image: fedora
    command: [/bin/bash]
    args: ["-c", "while true; do sleep 10; done"]
    volumeMounts:
    - name: my-first-linstor-volume
      mountPath: /data
    ports:
    - containerPort: 80
  volumes:
  - name: my-first-linstor-volume
    persistentVolumeClaim:
      claimName: "my-first-linstor-volume"

我们可以使用以下命令创建 Pod

kubectl create -f my-first-linstor-volume-pod.yaml

Running kubectl describe pod fedora can be used to confirm that Pod scheduling and volume attachment succeeded. Examining the PersistentVolumeClaim, we can see that it is now bound to a volume.

To remove a volume, please ensure that no pod is using it and then delete the PersistentVolumeClaim using the kubectl command. For example, to remove the volume that we just made, run the following two commands, noting that the Pod must be unscheduled before the PersistentVolumeClaim will be removed:

kubectl delete pod fedora # unschedule the pod.

kubectl get pod -w # wait for pod to be unscheduled

kubectl delete pvc my-first-linstor-volume # remove the PersistentVolumeClaim, the PersistentVolume, and the LINSTOR Volume.

3.4.1. Available Parameters in a Storage Class

The following storage class contains all currently available parameters to configure the provisioned storage.

linstor.csi.linbit.com/ is an optional, but recommended prefix for LINSTOR CSI specific parameters.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: full-example
provisioner: linstor.csi.linbit.com
parameters:
  # CSI related parameters
  csi.storage.k8s.io/fstype: xfs
  # LINSTOR parameters
  linstor.csi.linbit.com/autoPlace: "2"
  linstor.csi.linbit.com/placementCount: "2"
  linstor.csi.linbit.com/resourceGroup: "full-example"
  linstor.csi.linbit.com/storagePool: "my-storage-pool"
  linstor.csi.linbit.com/disklessStoragePool: "DfltDisklessStorPool"
  linstor.csi.linbit.com/layerList: "drbd storage"
  linstor.csi.linbit.com/placementPolicy: "AutoPlaceTopology"
  linstor.csi.linbit.com/allowRemoteVolumeAccess: "true"
  linstor.csi.linbit.com/encryption: "true"
  linstor.csi.linbit.com/nodeList: "diskful-a diskful-b"
  linstor.csi.linbit.com/clientList: "diskless-a diskless-b"
  linstor.csi.linbit.com/replicasOnSame: "zone=a"
  linstor.csi.linbit.com/replicasOnDifferent: "rack"
  linstor.csi.linbit.com/disklessOnRemaining: "false"
  linstor.csi.linbit.com/doNotPlaceWithRegex: "tainted.*"
  linstor.csi.linbit.com/fsOpts: "-E nodiscard"
  linstor.csi.linbit.com/mountOpts: "noatime"
  linstor.csi.linbit.com/postMountXfsOpts: "extsize 2m"
  # Linstor properties
  property.linstor.csi.linbit.com/*: <x>
  # DRBD parameters
  DrbdOptions/*: <x>

3.4.2. csi.storage.k8s.io/fstype

The csi.storage.k8s.io/fstype parameter sets the file system type to create for volumeMode: FileSystem PVCs. Currently supported are:

  • ext4 (default)

  • xfs

3.4.3. autoPlace

autoPlace is an integer that determines the amount of replicas a volume of this StorageClass will have. For instance, autoPlace: "3" will produce volumes with three-way replication. If neither autoPlace nor nodeList are set, volumes will be automatically placed on one node.

如果使用此选项,则不能使用nodeList
You have to use quotes, otherwise Kubernetes will complain about a malformed StorageClass.
此选项(以及影响自动放置行为的所有选项)修改将在其上配置卷的底层存储的LINSTOR节点的数量,并且与可从中访问这些卷的 kubelets 正交。

3.4.4. placementCount

placementCount is an alias for autoPlace

3.4.5. resourceGroup

The LINSTOR Resource Group (RG) to associate with this StorageClass. If not set, a new RG will be created for each new PVC.

3.4.6. storagePool

storagePool 是用于为新创建的卷提供存储的LINSTOR storage pool的名称。

只有配置了相同 storage pool 的节点才被考虑用于autoplacement。同样,对于使用nodeList该列表中指定的所有节点,都必须在其上配置此 storage pool

3.4.7. disklessStoragePool

disklessStoragePool 是一个可选参数,它只影响作为客户端无磁盘分配给 kubelets 的LINSTOR卷。如果在LINSTOR中定义了自定义 diskless storage pool,请在此处指定。

3.4.8. layerList

A comma-separated list of layers to use for the created volumes. The available layers and their order are described towards the end of this section. Defaults to drbd,storage

3.4.9. placementPolicy

Select from one of the available volume schedulers:

  • AutoPlaceTopology, the default: Use topology information from Kubernetes together with user provided constraints (see replicasOnSame and replicasOnDifferent).

  • AutoPlace Use LINSTOR autoplace, influenced by replicasOnSame and replicasOnDifferent

  • FollowTopology: Use CSI Topology information to place at least one volume in each “preferred” zone. Only useable if CSI Topology is enabled.

  • Manual: Use only the nodes listed in nodeList and clientList.

  • Balanced: EXPERIMENTAL Place volumes across failure domains, using the least used storage pool on each selected node.

3.4.10. allowRemoteVolumeAccess

Control on which nodes a volume is accessible. The value for this option can take two different forms:

  • A simple "true" or "false" allows access from all nodes, or only those nodes with diskfull resources.

  • Advanced rules, which allow more granular rules on which nodes can access the volume.

    The current implementation can grant access to the volume for nodes that share the same labels. For example, if you want to allow access from all nodes in the same region and zone as a diskfull resource, you could use:

    parameters:
      linstor.csi.linbit.com/allowRemoteVolumeAccess: |
        - fromSame:
          - topology.kubernetes.io/region
          - topology.kubernetes.io/zone

    You can specify multiple rules. The rules are additive, a node only need to match one rule to be assignable.

3.4.11. encryption

encryption is an optional parameter that determines whether to encrypt volumes. LINSTOR must be configured for encryption for this to work properly.

3.4.12. 节点列表

nodeList 是要分配给卷的节点列表。这将把卷分配给每个节点,并在所有节点之间进行复制。这也可以用于按主机名选择单个节点,但使用replicasOnSame选择单个节点更灵活。

如果使用此选项,则不能使用autoPlace
此选项确定卷的底层存储将在哪个LINSTOR节点上配置,并且与从那个 kubelets 可以访问这些卷位置正交。

3.4.13. clientList

clientList is a list of nodes for diskless volumes to be assigned to. Use in conjunction with 节点列表.

3.4.14. replicasOnSame

replicasOnSame is a list of key or key=value items used as autoplacement selection labels when autoplace is used to determine where to provision storage. These labels correspond to LINSTOR node properties.

The operator periodically synchronizes all labels from Kubernetes Nodes, so you can use them as keys for scheduling constraints.

Let’s explore this behavior with examples assuming a LINSTOR cluster such that node-a is configured with the following auxiliary property zone=z1 and role=backups, while node-b is configured with only zone=z1.

如果我们使用 autoPlace: "1" ` 和 `replicasOnSame: "zone=z1 role=backups"`来配置一个 StorageClass ,那么从该 StorageClass 创建的所有卷都将配置在 node-a 上,因为这是LINSTOR集群中唯一具有所有正确的key=value对的节点。这是选择单个节点进行资源调配的最灵活方式。

This guide assumes LINSTOR CSI version 0.10.0 or newer. All properties referenced in replicasOnSame and replicasOnDifferent are interpreted as auxiliary properties. If you are using an older version of LINSTOR CSI, you need to add the Aux/ prefix to all property names. So replicasOnSame: "zone=z1" would be replicasOnSame: "Aux/zone=z1" Using Aux/ manually will continue to work on newer LINSTOR CSI versions.

如果我们使用 autoPlace: "1" ` 和 `replicasOnSame: "zone=z1" ` 来配置一个 StorageClass ,那么卷将在 `node-anode-b 上配置,因为它们都有 zone=z1 辅助属性。

If we configure a StorageClass with autoPlace: "2" and replicasOnSame: "zone=z1 role=backups", then provisioning will fail, as there are not two or more nodes that have the appropriate auxiliary properties.

如果我们用 autoPlace: "2"replicasOnSame: "zone=z1" 配置一个存储类,那么卷将同时在 node-anode-b 上配置,因为它们都有 zone=z1 辅助属性。

You can also use a property key without providing a value to ensure all replicas are placed on nodes with the same property value, with caring about the particular value. Assuming there are 4 nodes, node-a1 and node-a2 are configured with zone=a. node-b1 and node-b2 are configured with zone=b. Using autoPlace: "2" and replicasOnSame: "zone" will place on either node-a1 and node-a2 OR on node-b1 and node-b2.

3.4.15. replicasOnDifferent

replicasOnDifferent takes a list of properties to consider, same as replicasOnSame. There are two modes of using replicasOnDifferent:

  • Preventing volume placement on specific nodes:

    If a value is given for the property, the nodes which have that property-value pair assigned will be considered last.

    Example: replicasOnDifferent: "no-csi-volumes=true" will place no volume on any node with property no-csi-volumes=true unless there are not enough other nodes to fulfill the autoPlace setting.

  • Distribute volumes across nodes with different values for the same key:

    If no property value is given, LINSTOR will place the volumes across nodes with different values for that property if possible.

    Example: Assuming there are 4 nodes, node-a1 and node-a2 are configured with zone=a. node-b1 and node-b2 are configured with zone=b. Using a StorageClass with autoPlace: "2" and replicasOnDifferent: "zone", LINSTOR will create one replica on either node-a1 or node-a2 and one replica on either node-b1 or node-b2.

3.4.16. disklessOnRemaining

Create a diskless resource on all nodes that were not assigned a diskful resource.

3.4.17. doNotPlaceWithRegex

Do not place the resource on a node which has a resource with a name matching the regex.

3.4.18. fsOpts

fsOpts 是一个可选参数,在创建时将选项传递给卷的文件系统。

请注意,这些值特定于您选择的filesystem

3.4.19. mountOpts

mountOpts 是一个可选参数,在装载时将选项传递给卷的文件系统。

3.4.20. postMountXfsOpts

Extra arguments to pass to xfs_io, which gets called before right before first use of the volume.

3.4.21. property.linstor.csi.linbit.com/*

Parameters starting with property.linstor.csi.linbit.com/ are translated to LINSTOR properties that are set on the Resource Group associated with the StorageClass.

For example, to set DrbdOptions/auto-quorum to disabled, use:

property.linstor.csi.linbit.com/DrbdOptions/auto-quorum: disabled

The full list of options is available here

3.4.22. DrbdOptions/*: <x>

This option is deprecated, use the more general property.linstor.csi.linbit.com/* form.

Advanced DRBD options to pass to LINSTOR. For example, to change the replication protocol, use DrbdOptions/Net/protocol: "A".

3.5. 快照

Creating snapshots and creating new volumes from snapshots is done using VolumeSnapshots, VolumeSnapshotClasses, and PVCs.

3.5.1. Adding Snapshot Support

LINSTOR supports the volume snapshot feature, which is configured in some, but not all Kubernetes distributions.

To check if your Kubernetes distribution supports snapshots out of the box, run the following command:

$ kubectl get --raw /apis/snapshot.storage.k8s.io/v1
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"snapshot.storage.k8s.io/v1"...
$ # If your distribution does NOT support snapshots out of the box, you will get a different response:
$ kubectl get --raw /apis/snapshot.storage.k8s.io/v1
Error from server (NotFound): the server could not find the requested resource

In case your Kubernetes distribution does not support snapshots, you can manually add the required components from the Kubernetes Storage SIG. For convenience, you can use Helm Charts provided by the Piraeus team to add these components.

Listing 5. Adding snapshot support using the Piraeus Charts
$ kubectl create namespace snapshot-controller
$ helm repo add piraeus-charts https://piraeus.io/helm-charts/
$ helm install -n snapshot-controller snapshot-validation-webhook \
  piraeus-charts/snapshot-validation-webhook
$ helm install -n snapshot-controller snapshot-controller \
  piraeus-charts/snapshot-controller --wait

3.5.2. Using Volume Snapshots

Then we can create our VolumeSnapshotClass:

Listing 6. my-first-linstor-snapshot-class.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: my-first-linstor-snapshot-class
driver: linstor.csi.linbit.com
deletionPolicy: Delete

使用 kubectl 创建 VolumeSnapshotClass

kubectl create -f my-first-linstor-snapshot-class.yaml

现在,我们将为上面创建的卷创建卷快照。这是用 VolumeSnapshot 完成的:

Listing 7. my-first-linstor-snapshot.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: my-first-linstor-snapshot
spec:
  volumeSnapshotClassName: my-first-linstor-snapshot-class
  source:
    persistentVolumeClaimName: my-first-linstor-volume

使用 kubectl 创建 VolumeSnapshot

kubectl create -f my-first-linstor-snapshot.yaml

You can check that the snapshot creation was successful

kubectl describe volumesnapshots.snapshot.storage.k8s.io my-first-linstor-snapshot
...
Spec:
  Source:
    Persistent Volume Claim Name:  my-first-linstor-snapshot
  Volume Snapshot Class Name:      my-first-linstor-snapshot-class
Status:
  Bound Volume Snapshot Content Name:  snapcontent-b6072ab7-6ddf-482b-a4e3-693088136d2c
  Creation Time:                       2020-06-04T13:02:28Z
  Ready To Use:                        true
  Restore Size:                        500Mi

最后,我们将使用 PVC 从快照创建一个新卷。

Listing 8. my-first-linstor-volume-from-snapshot.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-first-linstor-volume-from-snapshot
spec:
  storageClassName: linstor-basic-storage-class
  dataSource:
    name: my-first-linstor-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi

kubectl 创建 PVC

kubectl create -f my-first-linstor-volume-from-snapshot.yaml
Storing Snapshots on S3 Storage

LINSTOR can store snapshots on S3 compatible storage for disaster recovery. This is integrated in Kubernetes using special a special VolumeSnapshotClass:

---
kind: VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1
metadata:
  name: linstor-csi-snapshot-class-s3
driver: linstor.csi.linbit.com
deletionPolicy: Retain
parameters:
  snap.linstor.csi.linbit.com/type: S3
  snap.linstor.csi.linbit.com/remote-name: backup-remote
  snap.linstor.csi.linbit.com/allow-incremental: "false"
  snap.linstor.csi.linbit.com/s3-bucket: snapshot-bucket
  snap.linstor.csi.linbit.com/s3-endpoint: s3.us-west-1.amazonaws.com
  snap.linstor.csi.linbit.com/s3-signing-region: us-west-1
  snap.linstor.csi.linbit.com/s3-use-path-style: "false"
  # Refer here to the secret that holds access and secret key for the S3 endpoint.
  # See below for an example.
  csi.storage.k8s.io/snapshotter-secret-name: linstor-csi-s3-access
  csi.storage.k8s.io/snapshotter-secret-namespace: storage
---
kind: Secret
apiVersion: v1
metadata:
  name: linstor-csi-s3-access
  namespace: storage
immutable: true
type: linstor.csi.linbit.com/s3-credentials.v1
stringData:
  access-key: access-key
  secret-key: secret-key

Check the LINSTOR snapshot guide on the exact meaning of the snap.linstor.csi.linbit.com/ parameters. The credentials used to log in are stored in a separate secret, as show in the example above.

Referencing the above storage class when creating snapshots causes the snapshots to be automatically uploaded to the configured S3 storage.

Restoring from Pre-existing Snapshots

Restoring from pre-existing snapshots is an important step in disaster recovery. A snapshot needs to be registered with Kubernetes before it can be used to restore.

If the snapshot that should be restored is part of a backup to S3, the LINSTOR “remote” needs to be configured first.

linstor remote create s3 backup-remote s3.us-west-1.amazonaws.com \
  snapshot-bucket us-west-1 access-key secret-key
linstor backup list backup-remote

The snapshot you want to register needs to be one of the listed snapshots.

To register the snapshot with Kubernetes, you need to create two resources, one VolumeSnapshotContent referencing the ID of the snapshot and one VolumeSnapshot, referencing the content.

---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: example-backup-from-s3
  namespace: project
spec:
  source:
    volumeSnapshotContentName: restored-snap-content-from-s3
  volumeSnapshotClassName: linstor-csi-snapshot-class-s3
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotContent
metadata:
  name: restored-snap-content-from-s3
spec:
  deletionPolicy: Delete
  driver: linstor.csi.linbit.com
  source:
    snapshotHandle: snapshot-id
  volumeSnapshotClassName: linstor-csi-snapshot-class-s3
  volumeSnapshotRef:
    apiVersion: snapshot.storage.k8s.io/v1
    kind: VolumeSnapshot
    name: example-backup-from-s3
    namespace: project

Once applied, the VolumeSnapshot should be shown as ready, at which point you can reference it as a dataSource in a PVC.

3.6. Volume Accessibility and Locality

LINSTOR volumes are typically accessible both locally and over the network. The CSI driver will ensure that the volume is accessible on whatever node was selected for the consumer. The driver also provides options to ensure volume locality (the consumer is placed on the same node as the backing data) and restrict accessibility (only a subset of nodes can access the volume over the network).

Volume locality is achieved by setting volumeBindingMode: WaitForFirstConsumer in the storage class. This tell Kubernetes and the CSI driver to wait until the first consumer (Pod) referencing the PVC is scheduled. The CSI driver then provisions the volume with backing data on the same node as the consumer. In case a node without appropriate storage pool was selected, a replacement node in the set of accessible nodes is chosen (see below).

Volume accessibility is controlled by the allowRemoteVolumeAccess parameter. Whenever the CSI plug-in needs to place a volume, this parameter is consulted to get the set of “accessible” nodes. This means they can share volumes placed on them through the network. This information is also propagated to Kubernetes using label selectors on the PV.

3.6.1. Volume Accessibility and Locality Examples

The following example show common scenarios where you want to optimize volume accessibility and locality. It also includes examples of how to spread volume replicas across zones in a cluster.

Single-zone Homogeneous Clusters

The cluster only spans a single zone, so latency between nodes is low. The cluster is homogeneous, that is, all nodes are configured similarly. All nodes have their own local storage pool.

Listing 9. example-storage-class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-storage
provisioner: linstor.csi.linbit.com
volumeBindingMode: WaitForFirstConsumer (1)
parameters:
  linstor.csi.linbit.com/storagePool: linstor-pool (2)
  linstor.csi.linbit.com/placementCount: "2" (3)
  linstor.csi.linbit.com/allowRemoteVolumeAccess: "true" (4)
1 Enable late volume binding. This places one replica on the same node as the first consuming pod, if possible.
2 Set the storage pool(s) to use.
3 Ensure that the data is replicated, so that at least 2 nodes store the data.
4 Allow using the volume even on nodes without replica. Since all nodes are connected equally, performance impact should be manageable.
Multi-zonal Homogenous Clusters

As before, in our homogenous cluster all nodes are configured similarly with their own local storage pool. The cluster spans now multiple zones, with increased latency across nodes in different zones. To ensure low latency, we want to restrict access to the volume with a local replica to only those zones that do have a replica. At the same time, we want to spread our data across multiple zones.

Listing 10. example-storage-class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-storage
provisioner: linstor.csi.linbit.com
volumeBindingMode: WaitForFirstConsumer (1)
parameters:
  linstor.csi.linbit.com/storagePool: linstor-pool (2)
  linstor.csi.linbit.com/placementCount: "2" (3)
  linstor.csi.linbit.com/allowRemoteVolumeAccess: | (4)
    - fromSame:
      - topology.kubernetes.io/zone
  linstor.csi.linbit.com/replicasOnDifferent: topology.kubernetes.io/zone (5)
1 Enable late volume binding. This places one replica on the same node as the first consuming pod, if possible.
2 Set the storage pool(s) to use.
3 Ensure that the data is replicated, so that at least 2 nodes store the data.
4 Allow using the volume on nodes in the same zone as a replica, under the assumption that zone internal networking is fast and low latency.
5 Spread the replicas across different zones.
Multi-region Clusters

Our cluster now spans multiple regions. We don’t want to incur the latency penalty to replicate our data across regions, we just want to replicate in the same zone.

Listing 11. example-storage-class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-storage
provisioner: linstor.csi.linbit.com
volumeBindingMode: WaitForFirstConsumer (1)
parameters:
  linstor.csi.linbit.com/storagePool: linstor-pool (2)
  linstor.csi.linbit.com/placementCount: "2" (3)
  linstor.csi.linbit.com/allowRemoteVolumeAccess: | (4)
    - fromSame:
      - topology.kubernetes.io/zone
  linstor.csi.linbit.com/replicasOnSame: topology.kubernetes.io/region (5)
1 Enable late volume binding. This places one replica on the same node as the first consuming pod, if possible.
2 Set the storage pool(s) to use.
3 Ensure that the data is replicated, so that at least 2 nodes store the data.
4 Allow using the volume on nodes in the same zone as a replica, under the assumption that zone internal networking is fast and low latency.
5 Restrict replicas to only a single region.
Cluster with External Storage

Our cluster now only consists of compute nodes without local storage. Any volume access has to occur through remote volume access.

Listing 12. example-storage-class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-storage
provisioner: linstor.csi.linbit.com
parameters:
  linstor.csi.linbit.com/storagePool: linstor-pool (1)
  linstor.csi.linbit.com/placementCount: "1" (2)
  linstor.csi.linbit.com/allowRemoteVolumeAccess: "true" (3)
1 Set the storage pool(s) to use.
2 Assuming we only have one storage host, we can only place a single volume without additional replicas.
3 Our worker nodes need to be allowed to connect to the external storage host.

3.7. LINSTOR Affinity Controller

Volume Accessibility is controlled by the node affinity of the PersistentVolume (PV). This affinity is static, that is once defined it cannot be changed.

This can be an issue if you would like to use a strict affinity: Your PV is pinned to specific nodes, but you might want to remove or add nodes. While LINSTOR can move the volume (for example: this happens automatically if you remove a node in kubernetes), the PV affinity is not updated to reflect this.

This is where the LINSTOR Affinity Controller comes in: it watches PVs and compares their affinity with the volumes’ states in LINSTOR. If they go out of sync, the PV is replaced with an updated version.

The LINSTOR Affinity Controller is packaged in a Helm chart. If you install it in the same namespace as the Operator, simply run:

$ helm repo update
$ helm install linstor-affinity-controller linstor/linstor-affinity-controller

Additional options for the chart are available at the upstream project.

3.8. Volume Locality Optimization Using LINSTOR Scheduler

We maintain an open source plug-in for the Kubernetes scheduler. The scheduler will take the current placement of volumes into account and optimize for data locality. If possible, the pod will be assigned to a node that also hosts replicas of attached volumes, reducing latency for read operations.

The scheduler is available as a separate chart from artifacthub.io. The chart will deploy a new scheduler, which you can later use when creating pod resources:

apiVersion: v1
kind: Pod
metadata:
  name: busybox
spec:
  schedulerName: linstor-scheduler (1)
  containers:
  - name: busybox
    image: busybox
    command: ["tail", "-f", "/dev/null"]
    volumeMounts:
    - name: my-first-linstor-volume
      mountPath: /data
    ports:
    - containerPort: 80
  volumes:
  - name: my-first-linstor-volume
    persistentVolumeClaim:
      claimName: "test-volume"
1 Add the name of the scheduler to your pod.

3.9. Fast Workload Failover Using the High Availability Controller

When node failures occur, Kubernetes is very conservative in rescheduling stateful workloads. This means it can take more than 15 minutes for Pods to be moved from unreachable nodes. With the information available to DRBD and LINSTOR, this process can be sped up significantly.

The LINSTOR High Availability Controller (HA Controller) speeds up the failover process for stateful workloads using LINSTOR for storage. It monitors and manages any Pod that is attached to at least one DRBD resource.

For the HA Controller to work properly, you need quorum, that is at least three replicas (or two replicas + one diskless tiebreaker). If using lower replica counts, attached Pods will be ignored and are not eligible for faster failover.

The HA Controller is packaged as a Helm chart, and can be deployed using:

$ helm repo update
$ helm install linstor-ha-controller linstor/linstor-ha-controller

If you are using the HA Controller in your cluster you can set additional parameters in all StorageClasses. These parameters ensure that the volume is not accidentally remounted as read-only, leading to degraded Pods.

parameters:
  property.linstor.csi.linbit.com/DrbdOptions/auto-quorum: suspend-io
  property.linstor.csi.linbit.com/DrbdOptions/Resource/on-no-data-accessible: suspend-io
  property.linstor.csi.linbit.com/DrbdOptions/Resource/on-suspended-primary-outdated: force-secondary
  property.linstor.csi.linbit.com/DrbdOptions/Net/rr-conflict: retry-connect

To exempt a Pod from management by the HA Controller, add the following annotation to the Pod:

$ kubectl annotate pod <podname> drbd.linbit.com/ignore-fail-over=""

3.10. Evacuating a Node in Kubernetes

If you want to evacuate a LINSTOR node of its resources, so that they are placed onto other nodes within your cluster, the process is detailed in Evacuating a Node. However, before evacuating a LINSTOR node in Kubernetes, you need to take an additional action.

First move the node’s workload to another node. You can do this by entering the command:

# kubectl drain --ignore-daemonsets <node_name>

After verifying that your cluster is running all right, you can continue to follow the steps in Evacuating a Node.

If you are planning on evacuating more than one node, enter the following command on all the nodes that you will be evacuating:

# linstor node set-property n1.k8s-mwa.at.linbit.com AutoplaceTarget false

This ensures that LINSTOR will not place resources from a node that you are evacuating onto another node that you plan on evacuating.

3.11. Upgrading a LINSTOR Deployment on Kubernetes

A LINSTOR Deployment on Kubernetes can be upgraded to a new release using Helm.

Before upgrading to a new release, you should ensure you have an up-to-date backup of the LINSTOR database. If you are using the etcd database packaged in the LINSTOR Chart, see here

Upgrades using the LINSTOR etcd deployment require etcd to use persistent storage. Only follow these steps if etcd was deployed using etcd.persistentVolume.enabled=true

Upgrades will update to new versions of the following components:

  • LINSTOR Operator deployment

  • LINSTOR Controller

  • LINSTOR Satellite

  • LINSTOR CSI Driver

  • etcd

  • Stork

Some versions require special steps, please take a look here The main command to upgrade to a new LINSTOR Operator version is:

helm repo update
helm upgrade linstor-op linstor/linstor

If you used any customizations on the initial install, pass the same options to helm upgrade. The options currently in use can be retrieved from Helm.

# Retrieve the currently set options
$ helm get values linstor-op
USER-SUPPLIED VALUES:
USER-SUPPLIED VALUES: null
drbdRepoCred: drbdiocred
operator:
  satelliteSet:
    kernelModuleInjectionImage: drbd.io/drbd9-rhel8:v9.0.28
    storagePools:
      lvmThinPools:
      - devicePaths:
        - /dev/vdb
        name: thinpool
        thinVolume: thinpool
        volumeGroup: ""
# Save current options
$ helm get values linstor-op > orig.yaml
# modify values here as needed. for example selecting a newer DRBD image
$ vim orig.yaml
# Start the upgrade
$ helm upgrade linstor-op linstor/linstor -f orig.yaml

This triggers the rollout of new pods. After a short wait, all pods should be running and ready. Check that no errors are listed in the status section of LinstorControllers, LinstorSatelliteSets and LinstorCSIDrivers.

During the upgrade process, provisioning of volumes and attach/detach operations might not work. Existing volumes and volumes already in use by a pod will continue to work without interruption.

3.11.1. Upgrading a LINSTOR Cluster with Kubernetes Back End

The LINSTOR Operator will create a backup of the database before upgrading. This makes it possible to roll back to a known state should something go wrong during the upgrade.

There are situations in which the Operator can’t create the backup automatically. This might be because:

  • The base version of the chart or operator is too old. Automatic backups are available starting with version 1.8.0 If upgrading from a version before 1.8.0, please follow the manual steps in the next section.

  • The backup is too large to fit into a Kubernetes secret. In this case an error is reported in the status field of the LinstorController resources. Follow the instructions by copying the created backup to a safe location and creating the necessary secret.

    kubectl cp <linstor-operator-pod>:/run/linstor-backups/linstor-backup-<some-hash>.tar.gz <destination-path>
    kubectl create secret linstor-backup-<same-hash>
Creating a backup of the LINSTOR Database

Follow these instructions if you need to manually create a backup of the LINSTOR k8s database.

  1. Stop the current controller:

    $ kubectl patch linstorcontroller linstor-op-cs '{"spec":{"replicas": 0}}'
    $ kubectl rollout status --watch deployment/linstor-op-cs-controller
  2. The following command will create a file crds.yaml, which stores the current state of all LINSTOR Custom Resource Definitions:

    $ kubectl get crds | grep -o ".*.internal.linstor.linbit.com" | \
      xargs kubectl get crds -oyaml > crds.yaml
  3. In addition to the definitions, the actual resources must be backed up as well:

    $ kubectl get crds | grep -o ".*.internal.linstor.linbit.com" | \
      xargs -i{} sh -c "kubectl get {} -oyaml > {}.yaml"
  4. If upgrading the chart, use --set IHaveBackedUpAllMyLinstorResources=true to acknowledge you have executed the above steps.

Restoring from a LINSTOR Database backup

Follow these instructions if you need to recover from an failure during a LINSTOR upgrade.

  1. Fetch the backup (skip if the backup is already available on your local machine):

    $ # List the available backups
    $ kubectl get secret '-ocustom-columns=NAME:.metadata.name,FROM:metadata.annotations.linstor\.linbit\.com/backup-previous-version,CREATED-AT:.metadata.creationTimestamp'
    $ kubectl get secret linstor-backup-<backup-specific-hash> '-ogo-template=go-template={{index .data ".binaryData.backup.tar.gz" | base64decode}}' > linstor-backup.tar.gz
  2. Unpack the backup

    $ tar xvf linstor-backup.tar.gz
    crds.yaml
    ....
  3. Stop the current controller:

    $ kubectl patch linstorcontroller linstor-op-cs "{"spec":{"replicas": 0}}"
    $ kubectl rollout status --watch deployment/piraeus-op-cs-controller
  4. Delete existing resources

    $ kubectl get crds | grep -o ".*.internal.linstor.linbit.com" | xargs --no-run-if-empty kubectl delete crds
  5. Apply the old LINSTOR CRDs

    $ kubectl apply -f crds.yaml
  6. Apply the old LINSTOR resource state

    $ kubectl apply -f *.internal.linstor.linbit.com.yaml
  7. Re-apply the helm chart using the old LINSTOR version

    $ helm upgrade linstor-op charts/piraeus --set operator.controller.controllerImage=... --set operator.satelliteSet.satelliteImage=...

3.11.2. Upgrading Instructions for Specific Versions

Some versions require special steps, see below.

Upgrading to 1.10

Version 1.10 introduces an option to share DRBD configuration between host and container. If you need this option, you have to update the CRDs. As Helm does not upgrade CRDs on chart upgrade, instead please run:

$ helm repo update
$ helm pull linstor/linstor --untar
$ kubectl replace -f linstor/crds/
customresourcedefinition.apiextensions.k8s.io/linstorcontrollers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorcsidrivers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorsatellitesets.linstor.linbit.com replaced
Upgrading to 1.9

Version 1.9 disables the LINSTOR HA Controller deployment by default. The deployment has moved out of the LINSTOR Operator chart. If you want to keep using the old version, enable it again using this Helm command:

helm upgrade linstor-op linstor/linstor ... --set haController.enabled=true

If you are upgrading to v1.9 from v1.6 or earlier, you need to either:

  1. Create a master passphrase, before you upgrade:

    $ kubectl create secret generic linstor-pass --from-literal=MASTER_PASSPHRASE=<password>
  2. Or, upgrade to v1.7 first, and Helm will create a master passphrase for you automatically. You can view this passphrase later, by entering:

    $ kubectl get secret linstor-op-passphrase \
    -ogo-template='{{ .data.MASTER_PASSPHRASE | base64decode }}'
Upgrading to v1.8
This upgrade requires a complete rebuild of the K8s database, so upgrades might take longer than normal.

Version 1.8 introduces new options to centrally set the log level and number of worker threads for the CSI driver. If you need these options, you have to update the CRDs. As Helm does not upgrade CRDs on chart upgrade, instead please run:

$ helm repo update
$ helm pull linstor/linstor --untar
$ kubectl replace -f linstor/crds/
customresourcedefinition.apiextensions.k8s.io/linstorcontrollers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorcsidrivers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorsatellitesets.linstor.linbit.com replaced

In addition, 1.8 reworks the way SSL/TLS setups work. Please revisit the secure deployment section and work through these steps again.

If you are upgrading to v1.8 from v1.6 or earlier, you need to either:

  1. Create a master passphrase, before you upgrade:

    $ kubectl create secret generic linstor-pass --from-literal=MASTER_PASSPHRASE=<password>
  2. Or, upgrade to v1.7 first, and Helm will create a master passphrase for you automatically. You can view this passphrase later, by entering:

    $ kubectl get secret linstor-op-passphrase \
    -ogo-template='{{ .data.MASTER_PASSPHRASE | base64decode }}'
Upgrading to v1.7

No additional steps necessary.

Upgrading to v1.6

This versions introduces a new option to support Kubernetes distributions which use different state directories than the default of /var/lib/kubelet. A notable example is microk8s, which uses /var/snap/microk8s/common/var/lib/kubelet. To support this, a small addition to the LinstorCSIDriver CRD was necessary. As Helm does not upgrade CRDs on chart upgrade, instead please run:

$ helm repo update
$ helm pull linstor/linstor --untar
$ kubectl replace -f linstor/crds/
customresourcedefinition.apiextensions.k8s.io/linstorcontrollers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorcsidrivers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorsatellitesets.linstor.linbit.com replaced

If you do not apply the new CRDs, you will get errors like the following:

Error: UPGRADE FAILED: error validating "": error validating data: ValidationError(LinstorCSIDriver.spec): unknown field "kubeletPath" in com.linbit.linstor.v1.LinstorCSIDriver.spec

If you previously used the included snapshot-controller to process VolumeSnapshot resources, you should replace it with the new charts provided by the Piraeus project. The section on snapshots has been updated to include instructions on how you can add the snapshot-controller to your cluster.

Upgrading to v1.5

This version introduces a monitoring component for DRBD resources. This requires a new image and a replacement of the existing LinstorSatelliteSet CRD. Helm does not upgrade the CRDs on a chart upgrade, instead please run:

$ helm repo update
$ helm pull linstor/linstor --untar
$ kubectl replace -f linstor/crds/
customresourcedefinition.apiextensions.k8s.io/linstorcontrollers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorcsidrivers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorsatellitesets.linstor.linbit.com replaced

If you do not plan to use the provided monitoring you still need to apply the above steps, otherwise you will get an errors like the following

Error: UPGRADE FAILED: error validating "": error validating data: ValidationError(LinstorSatelliteSet.spec): unknown field "monitoringImage" in com.linbit.linstor.v1.LinstorSatelliteSet.spec
Some Helm versions fail to set the monitoring image even after replacing the CRDs. In that case, the in-cluster LinstorSatelliteSet will show an empty monitoringImage value. Edit the resource using kubectl edit linstorsatellitesets and set the value to drbd.io/drbd-reactor:v0.3.0 to enable monitoring.
Upgrading to v1.4

This version introduces a new default version for the etcd image, so take extra care that etcd is using persistent storage. Upgrading the etcd image without persistent storage will corrupt the cluster.

If you are upgrading an existing cluster without making use of new Helm options, no additional steps are necessary.

If you plan to use the newly introduced additionalProperties and additionalEnv settings, you have to replace the installed CustomResourceDefinitions with newer versions. Helm does not upgrade the CRDs on a chart upgrade

$ helm pull linstor/linstor --untar
$ kubectl replace -f linstor/crds/
customresourcedefinition.apiextensions.k8s.io/linstorcontrollers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorcsidrivers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorsatellitesets.linstor.linbit.com replaced
Upgrading to v1.3

No additional steps necessary.

Upgrading to v1.2

LINSTOR Operator v1.2 is supported on Kubernetes 1.17+. If you are using an older Kubernetes distribution, you may need to change the default settings, for example [the CSI provisioner](https://kubernetes-csi.github.io/docs/external-provisioner.html).

There is a known issue when updating the CSI components: the pods will not be updated to the newest image and the errors section of the LinstorCSIDrivers resource shows an error updating the DaemonSet. In this case, manually delete deployment/linstor-op-csi-controller and daemonset/linstor-op-csi-node. They will be re-created by the Operator.

3.11.3. Creating etcd Backups

To create a backup of the etcd database and store it on your control host, run:

kubectl exec linstor-op-etcd-0 -- etcdctl snapshot save /tmp/save.db
kubectl cp linstor-op-etcd-0:/tmp/save.db save.db

These commands will create a file save.db on the machine you are running kubectl from.

4. LINSTOR Volumes in OpenShift

This chapter describes the usage of LINBIT SDS in OpenShift as managed by the LINSTOR Operator and with volumes provisioned using the LINSTOR CSI plugin.

OpenShift is the official Red Hat developed and supported distribution of Kubernetes. The value of OpenShift is the strong integration of otherwise optional components, like network ingress and monitoring, all tied together with a Web UI to manage it all. LINSTOR Operator integrates with these components where possible.

4.1. Deploying LINBIT SDS on OpenShift

Deploying LINBIT SDS on OpenShift is similar to deploying LINBIT SDS on other Kubernetes clusters. As prerequisites, you need:

  • Helm installed.

  • Access to your OpenShift cluster through the oc utility.

  • Your LINBIT Customer Portal credentials for accessing drbd.io.

LINBIT’s container image repository (http://drbd.io) is only available to LINBIT customers or through LINBIT customer trial accounts. Contact LINBIT for information on pricing or to begin a trial. Alternatively, you may use LINSTOR SDS’ upstream project named Piraeus, without being a LINBIT customer.

First, create a new project for LINBIT SDS and configure your credentials for drbd.io (replace $USERNAME and $PASSWORD with your own LINBIT Customer Portal credentials):

$ oc new-project storage
Now using project "storage" on server ...
$ oc create secret docker-registry drbdiocred --docker-server=drbd.io --docker-username=$USERNAME --docker-password=$PASSWORD
secret/drbdiocred created

Then, create a file to override some of the Helm chart’s default values, named override.yaml:

Listing 13. override.yaml
global:
  setSecurityContext: false (1)
etcd:
  enabled: false (2)
operator:
  controller:
    dbConnectionURL: k8s (3)
  satelliteSet:
    kernelModuleInjectionImage: drbd.io/drbd9-rhel8:v9.1.6 (4)
1 Helm would set a SecurityContext for all Pods that would conflict with the SecurityContext.
2 Use of etcd, although still the default, will be phased out in favor of the k8s backend for LINSTOR. New installations should not use etcd.
3 Use the new k8s backend for LINSTOR. The LINSTOR cluster state is stored inside CustomResources in the Kubernetes API. No additional database setup required.
4 OpenShift runs on Red Hat Core OS, which is based on Red Hat Enterprise Linux 8 and uses the same Linux kernel.

Make additional modifications to override.yaml based on your preference. Possible options are discussed in the Kubernetes advanced deployment section.

Finally, deploy LINBIT SDS using Helm and wait until all Pods are ready:

$ helm install linstor-op linstor/linstor --values override.yaml
NAME: linstor-op
NAMESPACE: storage
STATUS: deployed
...
$ oc wait --for=condition=Ready pod --all
...
$ oc get pods
NAME                                         READY   STATUS    RESTARTS   AGE
linstor-op-cs-controller-6588445756-mh9vn    1/1     Running   0          5m48s
...

4.1.1. Using LINBIT SDS on OpenShift

The next steps are exactly the same as in Kubernetes:

4.1.2. Configuring LINBIT GUI Access

The LINSTOR container images come with the LINBIT GUI preinstalled. To expose it on your cluster, configure an OpenShift Route resource:

$ oc create route edge --service linstor-op-cs
$ oc get route
NAME           HOST/PORT                                  PATH  SERVICES       PORT           TERMINATION  WILDCARD
linstor-op-cs  linstor-op-cs-storage.apps.oc.example.com        linstor-op-cs  linstor-op-cs  edge         None

The GUI is now accessible at https://linstor-op-cs-storage.apps.oc.example.com/ui/

This may enable external access to LINBIT GUI and LINSTOR API. Ensure that only authorized users can access it, for example by requiring client side TLS certificates on the route.

4.1.3. Configuring Cluster Monitoring

OpenShift includes a fully configured monitoring stack. While most of the monitoring stack is only intended for Openstack infrastructure, basic monitoring for LINBIT SDS is possible.

First, ensure that monitoring of user-defined projects is enabled in OpenShift by following the steps in Red Hat documentation.

Once monitoring for user-defined projects is enabled, the LINSTOR Operator automatically detects the presence of OpenShift’s Prometheus-based monitoring stack and configures the ServiceMonitor resources. The Prometheus instance will scrape metrics for DRBD and LINSTOR from all Cluster nodes.

4.2. Interacting with LINBIT SDS in OpenShift

The LINSTOR Controller pod includes a LINSTOR Client, making it easy to access LINSTOR’s interactive mode. For instance:

$ oc exec -it deployment/linstor-op-cs-controller -- linstor interactive
LINSTOR ==> ...

This should only be necessary for investigating problems and accessing advanced functionality. Regular operation such as creating volumes should be achieved through the Kubernetes integration.

5. LINSTOR Volumes in Nomad

This chapter describes using LINSTOR and DRBD to provision volumes in Nomad.

5.1. Introduction to Nomad

Nomad is a simple and flexible workload orchestrator to deploy and manage containers and non-containerized applications across on-premises and cloud environments.

Nomad supports provisioning storage volumes using plug-ins conforming to the Container Storage Interface (CSI).

LINBIT distributes a CSI plug-in in the form of container images from drbd.io. The plug-in can be configured to work with a LINSTOR cluster that is deployed along or inside a Nomad cluster.

LINBIT’s container image repository (http://drbd.io) is only available to LINBIT customers or through LINBIT customer trial accounts. Contact LINBIT for information on pricing or to begin a trial. Alternatively, you may use LINSTOR SDS’ upstream project named Piraeus, without being a LINBIT customer.

5.2. Deploying LINSTOR on Nomad

This section describes how you can deploy and configure a new LINSTOR cluster in Nomad.

If you want to install LINSTOR directly on your nodes, check out the guide on installing LINSTOR. You can skip this section and jump directly to deploying the CSI driver.

5.2.1. Preparing Nomad

To run LINSTOR, every Nomad agent needs to be configured to:

  • Support the docker driver and allow executing privileged containers

    To allow running privileged containers, add the following snippet to your Nomad agent configuration and restart Nomad

    Listing 14. /etc/nomad.d/docker-privileged.hcl
    plugin "docker" {
      config {
        allow_privileged = true
      }
    }
  • Support for container networking. If you don’t have the Container Network Interface plug-ins installed, you will only be able to use mode = "host" in your job networks. For most production setups, we recommend installing the default plug-ins:

    Head to the plug-in release page, select the release archive appropriate for your distribution and unpack them in /opt/cni/bin. You might need to create the directory before unpacking.

  • Provide a host volume, allowing a container access to the hosts /dev directory

    To create a host volume, add the following snippet to your Nomad agent configuration and restart Nomad.

    Listing 15. /etc/nomad.d/host-volume-dev.hcl
    client {
      host_volume "dev" {
        path = "/dev"
      }
    }

5.2.2. Creating a LINSTOR Controller Job

The LINSTOR Controller is deployed as a service with no replicas. At any point in time, there can only be one LINSTOR Controller running in a cluster. It is possible to restart the controller on a new node, provided it still has access to the database. See Creating a Highly Available LINSTOR Cluster for more information.

The following example will create a Nomad job starting a single LINSTOR Controller in datacenter dc1 and connect to an external database.

Listing 16. linstor-controller.hcl
job "linstor-controller" {
  datacenters = ["dc1"] (1)
  type = "service"

  group "linstor-controller" {
    network {
      mode = "bridge"
      # port "linstor-api" { (2)
      #   static = 3370
      #   to = 3370
      # }
    }

    service { (3)
      name = "linstor-api"
      port = "3370"

      connect {
        sidecar_service {}
      }

      check {
        expose = true
        type = "http"
        name = "api-health"
        path = "/health"
        interval = "30s"
        timeout = "5s"
      }
    }

    task "linstor-controller" {
      driver = "docker"
      config {
        image = "drbd.io/linstor-controller:v1.13.0" (4)

        auth { (5)
          username = "example"
          password = "example"
          server_address = "drbd.io"
        }

        mount {
          type = "bind"
          source = "local"
          target = "/etc/linstor"
        }
      }

      # template { (6)
      #  destination = "local/linstor.toml"
      #  data = <<EOH
      #    [db]
      #    user = "example"
      #    password = "example"
      #    connection_url = "jdbc:postgresql://postgres.internal.example.com/linstor"
      #  EOH
      # }

      resources {
        cpu    = 500 # 500 MHz
        memory = 700 # 700MB
      }
    }
  }
}
1 Replace dc1 with your own data center name
2 This exposes the LINSTOR API on the host on port 3370.
Uncomment this section if your cluster is not configured with Consul Connect.
3 The service block is used to expose the LINSTOR API to other jobs through the service mesh.
If your cluster is not configured for Consul Connect you can remove this section.
4 This sets the LINSTOR Controller image to run. The latest images are available from drbd.io.
The use of the :latest tag is strongly discouraged, as it can quickly lead to version mismatches and unintended upgrades.
5 Sets the authentication to use when pulling the image. If pulling from drbd.io, you need to use your LINBIT customer login here. Read more about pulling from a private repo here.
6 This template can be used to set arbitrary configuration options for LINSTOR. This example configures an external database for LINSTOR. You can find a more detailed explanation of LINSTOR’s database options here and more on Nomad templates here.

Apply the job by running:

$ nomad job run linstor-controller.hcl
==> Monitoring evaluation "7d8911a7"
    Evaluation triggered by job "linstor-controller"
==> Monitoring evaluation "7d8911a7"
    Evaluation within deployment: "436f4b2d"
    Allocation "0b564c73" created: node "07754224", group "controller"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "7d8911a7" finished with status "complete"
Using a Host Volume for LINSTOR’s Database

If you want to try LINSTOR without setting up an external database, you can make use of LINSTOR’s built-in filesystem based database. To make the database persistent, you need to ensure it is placed on a host volume.

Using a host volume means that only a single node is able to run the LINSTOR Controller. If the node is unavailable, the LINSTOR Cluster will also be unavailable. For alternatives, use an external (highly available) database or deploy the LINSTOR cluster directly on the hosts.

To create a host volume for the LINSTOR database, first create the directory on the host with the expected permissions

$ mkdir -p /var/lib/linstor
$ chown -R 1000:0 /var/lib/linstor

Then add the following snippet to your Nomad agent configuration and restart Nomad

Listing 17. /etc/nomad.d/host-volume-linstor-db.hcl
client {
  host_volume "linstor-db" {
    path = "/var/lib/linstor"
  }
}

Then, add the following snippets to the linstor-controller.hcl example from above and adapt the connection_url option from the configuration template.

Listing 18. job > group
volume "linstor-db" {
  type = "host"
  source = "linstor-db"
}
Listing 19. job > group > task
volume_mount {
  volume = "linstor-db"
  destination = "/var/lib/linstor"
}

template {
  destination = "local/linstor.toml"
  data = <<EOH
    [db]
    user = "linstor"
    password = "linstor"
    connection_url = "jdbc:h2:/var/lib/linstor/linstordb"
  EOH
}

5.2.3. Creating a LINSTOR Satellite Job

In Nomad, the LINSTOR satellites are deployed as a system job that runs in a privileged container. In addition to the satellites, the job will also load the DRBD module along with other kernel modules used by LINSTOR.

The following example will create a Nomad job starting a LINSTOR satellite on every node in data center dc1.

Listing 20. linstor-satellite.hcl
job "linstor-satellite" {
  datacenters = ["dc1"] (1)
  type = "system"

  group "satellite" {
    network {
      mode = "host"
    }

    volume "dev" { (2)
      type = "host"
      source = "dev"
    }

    task "linstor-satellite" {
      driver = "docker"

      config {
        image = "drbd.io/linstor-satellite:v1.13.0" (3)

        auth { (4)
          username = "example"
          password = "example"
          server_address = "drbd.io"
        }

        privileged = true (5)
        network_mode = "host" (6)
      }

      volume_mount { (2)
        volume = "dev"
        destination = "/dev"
      }

      resources {
        cpu    = 500 # 500 MHz
        memory = 500 # 500MB
      }
    }

    task "drbd-loader" {
      driver = "docker"
      lifecycle {
        hook = "prestart" (7)
      }

      config {
        image = "drbd.io/drbd9-rhel8:v9.0.29" (8)

        privileged = true (5)
        auth { (4)
          username = "example"
          password = "example"
          server_address = "drbd.io"
        }
      }

      env {
        LB_HOW = "shipped_modules" (9)
      }

      volume_mount { (10)
        volume = "kernel-src"
        destination = "/usr/src"
      }
      volume_mount { (10)
        volume = "modules"
        destination = "/lib/modules"
      }
    }

    volume "modules" { (10)
      type = "host"
      source = "modules"
      read_only = true
    }

    volume "kernel-src" { (10)
      type = "host"
      source = "kernel-src"
      read_only = true
    }
  }
}
1 Replace dc1 with your own data center name.
2 The dev host volume is the volume created in Preparing Nomad, which allows the satellite to manage the hosts block devices.
3 This sets the LINSTOR Satellite image to run. The latest images are available from drbd.io. The satellite image version has to match the version of the controller image.
The use of the :latest tag is strongly discouraged, as it can quickly lead to version mismatches and unintended upgrades.
4 Sets the authentication to use when pulling the image. If pulling from drbd.io, you need to use your LINBIT customer login here. Read more about pulling from a private repo here.
5 To configure storage devices, DRBD and load kernel modules, the containers need to be running in privileged mode.
6 The satellite needs to communicate with DRBD, which requires access to the netlink interface running in the hosts network.
7 The drbd-loader task will be executed once at the start of the satellite and load DRBD and other useful kernel modules.
8 The drbd-loader is specific to the distribution you are using. Available options are:
  • drbd.io/drbd9-bionic for Ubuntu 18.04 (Bionic Beaver)

  • drbd.io/drbd9-focal for Ubuntu 20.04 (Focal Fossa)

  • drbd.io/drbd9-rhel8 for RHEL 8

  • drbd.io/drbd9-rhel7 for RHEL 7

9 The drbd-loader container can be configured using environment variables. LB_HOW tells the container how to insert the DRBD kernel module. Available options are:
shipped_modules

Uses the prepackaged RPMs or DEBs delivered with the container.

compile

Compile DRBD from source. Requires access to the kernel headers (see below).

deps_only

Only try to load existing modules used by the LINSTOR satellite (for example dm_thin_pool and dm_cache).

10 In order for the drbd-loader container to build DRBD or load existing modules, it needs access to a hosts /usr/src and /lib/modules respectively.

This requires setting up additional host volumes on every node. The following snippet needs to be added to every Nomad agent confiration, then Nomad needs to be restarted.

Listing 21. /etc/nomad.d/drbd-loader-volumes.hcl
client {
  host_volume "modules" {
    path = "/lib/modules"
    read_only = true
  }
  host_volume "kernel-src" {
    path = "/usr/src"
    read_only = true
  }
}

Apply the job by running:

$ nomad job run linstor-satellite.hcl
==> Monitoring evaluation "0c07469d"
    Evaluation triggered by job "linstor-satellite"
==> Monitoring evaluation "0c07469d"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "0c07469d" finished with status "complete"

5.2.4. Configuring LINSTOR in Nomad

Once the linstor-controller and linstor-satellite jobs are running, you can start configuring the cluster using the linstor command line tool.

This can done:

  • directly by nomad exec-ing into the linstor-controller container.

  • using the drbd.io/linstor-client container on the host running the linstor-controller:

    docker run -it --rm --net=host drbd.io/linstor-client node create
  • by installing the linstor-client package on the host running the linstor-controller.

In all cases, you need to add the satellites to your cluster and create some storage pools. For example, to add the node nomad-01.example.com and configure a LVM Thin storage pool, you would run:

$ linstor node create nomad-01.example.com
$ linstor storage-pool create lvmthin nomad-01.example.com thinpool linstor_vg/thinpool
The CSI driver requires your satellites to be named after their hostname. To be precise, the satellite name needs to match Nomads attr.unique.hostname attribute on the node.

5.3. Deploying the LINSTOR CSI Driver in Nomad

The CSI driver is deployed as a system job, meaning it runs on every node in the cluster.

The following example will create a Nomad job starting a LINSTOR CSI Driver on every node in data center dc1.

Listing 22. linstor-csi.hcl
job "linstor-csi" {
  datacenters = ["dc1"] (1)
  type = "system"

  group "csi" {
    network {
      mode = "bridge"
    }

    service {
      connect {
        sidecar_service { (2)
          proxy {
            upstreams {
              destination_name = "linstor-api"
              local_bind_port  = 8080
            }
          }
        }
      }
    }

    task "csi-plugin" {
      driver = "docker"
      config {
        image = "drbd.io/linstor-csi:v0.13.1" (3)

        auth { (4)
          username = "example"
          password = "example"
          server_address = "drbd.io"
        }

        args = [
          "--csi-endpoint=unix://csi/csi.sock",
          "--node=${attr.unique.hostname}", (5)
          "--linstor-endpoint=http://${NOMAD_UPSTREAM_ADDR_linstor_api}", (6)
          "--log-level=info"
        ]

        privileged = true (7)
      }

      csi_plugin { (8)
        id = "linstor.csi.linbit.com"
        type = "monolith"
        mount_dir = "/csi"
      }

      resources {
        cpu    = 100 # 100 MHz
        memory = 200 # 200MB
      }
    }
  }
}
1 Replace dc1 with your own data center name
2 The sidecar_service stanza enables use of the service mesh generated by using Consul Connect. If you have not configured this feature in Nomad, or you are using an external LINSTOR Controller, you can skip this configuration.
3 This sets the LINSTOR CSI Driver image to run. The latest images are available from drbd.io.
The use of the :latest tag is strongly discouraged, as it can quickly lead to version mismatches and unintended upgrades.
4 Sets the authentication to use when pulling the image. If pulling from drbd.io, you need to use your LINBIT customer login here. Read more about pulling from a private repo here.
5 This argument sets the node name used by the CSI driver to identify itself in the LINSTOR API. By default, this is set to the node’s hostname.
6 This argument sets the LINSTOR API endpoint. If you are not using the consul service mesh (see Nr. 2 above), this needs to be set to the Controllers API endpoint. The endpoint needs to be reachable from every node this is deployed on.
7 The CSI driver needs to execute mount commands, requiring privileged containers.
8 The csi_plugin stanza informs Nomad that this task is a CSI plug-in. The Nomad agent will forward requests for volumes to one of the jobs containers.

Apply the job by running:

$ nomad job run linstor-csi.hcl
==> Monitoring evaluation "0119f19c"
    Evaluation triggered by job "linstor-csi"
==> Monitoring evaluation "0119f19c"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "0119f19c" finished with status "complete"

5.4. Using LINSTOR Volumes in Nomad

Volumes in Nomad are created using a volume-specification.

As an example, the following specification requests a 1GiB volume with 2 replicas from the LINSTOR storage pool thinpool.

Listing 23. vol1.hcl
id = "vol1" (1)
name = "vol1" (2)

type = "csi"
plugin_id = "linstor.csi.linbit.com"

capacity_min = "1GiB"
capacity_max = "1GiB"

capability {
  access_mode = "single-node-writer" (3)
  attachment_mode = "file-system" (4)
}

mount_options {
  fs_type = "ext4" (5)
}

parameters { (6)
  "resourceGroup" = "default-resource",
  "storagePool" = "thinpool",
  "autoPlace" = "2"
}
1 The id is used to reference this volume in Nomad. Used in the volume.source field of a job specification.
2 The name is used when creating the volume in the back end (that is, LINSTOR). Ideally this matches the id and is a valid LINSTOR resource name. If the name would not be valid, LINSTOR CSI will generate a random compatible name.
3 What kind of access the volume should support. LINSTOR CSI supports:
single-node-reader-only

Allow read only access on one node at a time.

single-node-writer

Allow read and write access on one node at a time.

multi-node-reader-only

Allow read only access from multiple nodes.

4 Can be file-system or block-device.
5 Specify the file system to use. LINSTOR CSI supports ext4 and xfs.
6 Additional parameters to pass to LINSTOR CSI. The example above requests the resource be part of the default-resource resource group and should deploy 2 replicas.

For a complete list of available parameters, you can check out the guide on Kubernetes storage classes. Kubernetes, like Nomad, makes use of the CSI plug-in.

To create the volume, run the following command:

$ nomad volume create vol1.hcl
Created external volume vol1 with ID vol1
$ nomad volume status
Container Storage Interface
ID    Name  Plugin ID               Schedulable  Access Mode
vol1  vol1  linstor.csi.linbit.com  true         <none>
$ linstor resource list
╭──────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node                 ┊ Port ┊ Usage  ┊ Conns ┊    State ┊ CreatedOn           ┊
╞══════════════════════════════════════════════════════════════════════════════════════════════╡
┊ vol1         ┊ nomad-01.example.com ┊ 7000 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2021-06-15 14:56:32 ┊
┊ vol1         ┊ nomad-02.example.com ┊ 7000 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2021-06-15 14:56:32 ┊
╰──────────────────────────────────────────────────────────────────────────────────────────────╯

5.4.1. Using Volumes in Jobs

To use the volume in a job, add the volume and volume_mount stanzas to the job specification:

job "example" {
  ...

  group "example" {
    volume "example-vol" {
      type = "csi"
      source = "vol1"
      attachment_mode = "file-system"
      access_mode = "single-node-writer"
    }

    task "mount-example" {
      volume_mount {
        volume = "example-vol"
        destination = "/data"
      }

      ...
    }
  }
}

5.4.2. Creating Snapshots of Volumes

LINSTOR can create snapshots of existing volumes, provided the underlying storage pool driver supports snapshots.

The following command creates a snapshot named snap1 of the volume vol1.

$ nomad volume snapshot create vol1 snap1
Snapshot ID  Volume ID  Size     Create Time  Ready?
snap1        vol1       1.0 GiB  None         true
$ linstor s l
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ SnapshotName ┊ NodeNames                                  ┊ Volumes  ┊ CreatedOn           ┊ State      ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ vol1         ┊ snap1        ┊ nomad-01.example.com, nomad-02.example.com ┊ 0: 1 GiB ┊ 2021-06-15 15:04:10 ┊ Successful ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

You can use a snapshot to pre-populate an existing volume with data from the snapshot

$ cat vol2.hcl
id = "vol2"
name = "vol2"
snapshot_id = "snap1"

type = "csi"
plugin_id = "linstor.csi.linbit.com"
...

$ nomad volume create vol2.hcl
Created external volume vol2 with ID vol2

6. Proxmox VE中的LINSTOR卷

This chapter describes DRBD in Proxmox Virtual Environment (VE) using the LINSTOR Proxmox Plug-in.

6.1. Introduction to Proxmox VE

proxmox VE是一个易于使用的、完整的服务器虚拟化环境,具有KVM、Linux容器和HA。

‘linstor-proxmox’ is a Perl plug-in for Proxmox that, in combination with LINSTOR, allows you to replicate VM disks on several Proxmox VE nodes. This allows to live-migrate active VMs within a few seconds and with no downtime without needing a central SAN, as the data is already replicated to multiple nodes.

6.2. Upgrading Proxmox

If this is a fresh installation, skip this section and continue with Installing the LINSTOR Proxmox Plug-in.

6.2.1. Upgrading Plug-in to 7.x

Version 7 of the plugin uses a LINSTOR controller API that is available from LINSTOR version 1.21.1 onwards. Make sure that your LINSTOR setup (controller and satellites) use at least that version.

6.2.2. Upgrading Plug-in from 4.x to 5.x

Version 5 of the plug-in drops compatibility with the legacy configuration options “storagepool” and “redundancy”. Version 5 requires a “resourcegroup” option, and obviously a LINSTOR resource group. The old options should be removed from the configuration.

Configuring LINSTOR is described in Section Configuring LINSTOR, a typical example follows:

Let’s assume the pool was set to “mypool”, and redundancy to 3.

# linstor resource-group create --storage-pool=mypool --place-count=3 drbdMypoolThree
# linstor volume-group create drbdMypoolThree
# vi /etc/pve/storage.cfg
drbd: drbdstorage
   content images,rootdir
   controller 10.11.12.13
   resourcegroup drbdMypoolThree

6.2.3. Upgrading Plug-in from 5.x to 6.x

Version 6.0.0 of the plug-in drops all code related to the redundancy setting. This is handled by LINSTOR resource groups (resourcegroup setting) for a very long time. No change should be required.

The controllervm setting, which was intended for executing a LINSTOR controller in a VM manged by LINSTOR is gone. Using drbd-reactor to realize a highly available LINSTOR controller is what we suggest.

The settings statuscache and preferlocal are now enabled by default.

6.2.4. Upgrading PVE from 5.x to 6.x

With version 6 PVE added additional parameters to some functions and rightfully reset their “APIAGE”. This means that old plug-ins, while actually usable as they don’t use any of these changed functions do not work anymore. Please upgrade to plug-in version 5.2.1 at least.

6.3. Installing the LINSTOR Proxmox Plug-in

To use LINSTOR in Proxmox, you will need to install the LINSTOR Proxmox plug-in.

6.3.1. Installing the Proxmox VE Kernel Headers

To use LINSTOR in Proxmox, you will need to install the DRBD kernel module. The DRBD 9 kernel module is installed as a kernel module source package (drbd-dkms). Therefore, you will have to install the Proxmox VE kernel headers package, pve-headers, before you install the DRBD kernel module from LINBIT’s repositories. Following that order ensures that the kernel module will build properly for your kernel.

If you do not plan to install the latest Proxmox kernel, you have to install kernel headers matching your current running kernel (for example, pve-headers-$(uname -r)). If you missed this step, then you can still rebuild the drbd-dkms package against your current kernel (so long as you have installed kernel headers in advance) by entering the apt install --reinstall drbd-dkms command.

You will need to add the Proxmox PVE repository to your APT sources list, /etc/apt/sources.list, and then enter apt update, before you can install the pve-headers package. Refer to the Proxmox wiki for instructions.

6.3.2. Installing the Proxmox Plug-in By Using LINBIT Customer Repositories

If you are a LINBIT customer, or you have an evaluation account, you can enable LINBIT’s drbd-9 repository on your node and then update your repositories by using an apt update command.

You can then install the DRBD kernel module, DRBD utilities, and the LINSTOR Proxmox plug-in by entering:

# apt install drbd-utils linstor-proxmox drbd-dkms
Refer to the Using a Script to Manage LINBIT Cluster Nodes for instructions on registering a node with LINBIT and enabling LINBIT repositories.

6.3.3. Installing the Proxmox Plug-in By Using LINBIT Public Repositories

LINBIT provides a dedicated public repository for Proxmox VE users. This repository not only contains the Proxmox plug-in, but the whole DRBD SDS stack including a DRBD SDS kernel module and user-space utilities.

LINBIT’s repository can be enabled as follows, where “$PVERS” should be set to your Proxmox VE major version (for example, “7”, not “7.1”):

# wget -O- https://packages.linbit.com/package-signing-pubkey.asc | apt-key add -
# PVERS=7 && echo "deb http://packages.linbit.com/public/ proxmox-$PVERS drbd-9" > \
/etc/apt/sources.list.d/linbit.list
# apt update && apt -y install linstor-proxmox

6.4. Configuring LINSTOR

For the rest of this guide we assume that you have a LINSTOR cluster configured as described in Initializing Your Cluster. Start the “linstor-controller” on one node, and the “linstor-satellite” on all nodes. The “linstor-satellite” service needs some extra configuration which should be done via systemctl edit linstor-satellite.service:

[Service]
Type=notify
TimeoutStartSec=infinity

The preferred way to use the plug-in, starting from version 4.1.0, is through LINSTOR resource groups and a single volume group within every resource group. LINSTOR resource groups are described in Using Resource Groups to Deploy LINSTOR Provisioned Volumes. All the required LINSTOR configuration (e.g., redundancy count) has to be set on the resource group.

6.5. Configuring the Proxmox Plug-in

最后一步是为Proxmox本身提供配置。这可以通过在 /etc/pve/storage.cfg 文件中添加一个条目来完成,其内容类似于以下内容。

drbd: drbdstorage
   content images,rootdir
   controller 10.11.12.13
   resourcegroup defaultpool

The “drbd” entry is fixed and you are not allowed to modify it, as it tells to Proxmox to use DRBD as storage back end. The “drbdstorage” entry can be modified and is used as a friendly name that will be shown in the PVE web GUI to locate the DRBD storage. The “content” entry is also fixed, so do not change it. The redundancy (specified in the resource group) specifies how many replicas of the data will be stored in the cluster. The recommendation is to set it to 2 or 3 depending on your setup. The data is accessible from all nodes, even if some of them do not have local copies of the data. For example, in a 5 node cluster, all nodes will be able to access 3 copies of the data, no matter where they are stored in. The “controller” parameter must be set to the IP of the node that runs the LINSTOR controller service. Only one node can be set to run as LINSTOR controller at the same time. If that node fails, start the LINSTOR controller on another node and change that value to its IP address.

A configuration using different storage pools in different resource groups would look like this:

drbd: drbdstorage
   content images,rootdir
   controller 10.11.12.13
   resourcegroup defaultpool

drbd: fastdrbd
   content images,rootdir
   controller 10.11.12.13
   resourcegroup ssd

drbd: slowdrbd
   content images,rootdir
   controller 10.11.12.13
   resourcegroup backup

By now, you should be able to create VMs using Proxmox’s web GUI by selecting “drbdstorage“, or any other of the defined pools as storage location.

Starting from version 5 of the plug-in, you can set the option “preferlocal yes”. If it is set, the plug-in tries to create a diskful assignment on the node that issued the storage create command. With this option you can ensure that the VM gets local storage if possible. Without that option LINSTOR might place the storage on nodes ‘B’ and ‘C’, while the VM is initially started on node ‘A’. This would still work as node ‘A’ then would get a diskless assignment, but having local storage might be preferred.

注意:目前DRBD只支持 raw 磁盘格式。

At this point you can try to live migrate the VM – as all data is accessible on all nodes (even on Diskless nodes) – it will take just a few seconds. The overall process might take a bit longer if the VM is under load and if there is a significant amount of RAM being dirtied all the time. But in any case, the downtime should be minimal and you will see no interruption at all.

Table 1. Table Configuration Options
Option Meaning

controller

The IP of the LINSTOR controller (‘,’ separated list allowed)

resourcegroup

The name of a LINSTOR resource group which defines the deployment of new VMs. As described above

preferlocal

Prefer to create local storage (yes/no). As decribed above

statuscache

Time in seconds status information is cached, 0 means no extra cache. Relevant on huge clusters with hundreds of resources. This has to be set on all drbd storages in /etc/pve/storage.cfg to take effect.

apicrt

Path to the client certificate

apikey

Path to the client private key

apica

Path to the CA certificate

6.6. Making the Controller Highly Available (Optional Configuration)

Making LINSTOR highly available is a matter of making the LINSTOR controller highly-available. This step is described in Section Creating a Highly Available LINSTOR Cluster.

The last — but crucial — step is to configure the Proxmox plug-in to be able to connect to multiple LINSTOR controllers. It will use the first one it receives an answer from. This is done by adding a comma-separated list of controllers in the controller section of the plug-in like this:

drbd: drbdstorage
   content images,rootdir
   controller 10.11.12.13,10.11.12.14,10.11.12.15
   resourcegroup defaultpool

7. OpenNebula中的LINSTOR卷

This chapter describes DRBD in OpenNebula using the LINSTOR storage driver add-on.

详细的安装和配置说明,请参见驱动程序源的 README.md 文件。

7.1. Introduction to OpenNebula

OpenNebula is a flexible and open source cloud management platform which allows its functionality to be extended using add-ons.

The LINSTOR add-on allows the deployment of virtual machines with highly available images backed by DRBD and attached across the network through DRBD’s own transport protocol.

7.2. Installing the OpenNebula Add-on

Installation of the LINSTOR storage add-on for OpenNebula requires a working OpenNebula cluster as well as a working LINSTOR cluster.

With access to LINBIT’s customer repositories you can install the linstor-opennebula package with:

# apt install linstor-opennebula

# yum install linstor-opennebula

Without access to LINBIT’s prepared packages you need to fall back to instructions on the OpenNebula LINSTOR Add-on GitHub page.

可以按照指南中的说明安装和配置带有LINSTOR的DRBD集群,请参见Initializing Your Cluster

OpenNebula和DRBD集群可以相互独立,但以下情况除外:OpenNebula的前端和主机节点必须包含在这两个集群中。

Host nodes do not need a local LINSTOR storage pool, as virtual machine images are attached to them across the network [1].

7.3. 部署选项

建议使用LINSTOR资源组配置您喜欢的部署,请参见Creating Resource Groups。不推荐使用以前的自动放置和部署节点模式, 因为已经过时了。

7.4. Configuring the OpenNebula Add-on

7.4.1. Adding the Driver to OpenNebula

修改 /etc/one/oned.conf 的以下部分`

将linstor添加到 TM MADDATASTORE MAD 部分的驱动程序列表中:

TM_MAD = [
    EXECUTABLE = "one_tm",
    ARGUMENTS = "-t 15 -d dummy,lvm,shared,fs_lvm,qcow2,ssh,vmfs,ceph,linstor"
]
DATASTORE_MAD = [
    EXECUTABLE = "one_datastore",
    ARGUMENTS  = "-t 15 -d dummy,fs,lvm,ceph,dev,iscsi_libvirt,vcenter,linstor -s shared,ssh,ceph,fs_lvm,qcow2,linstor"
]
linstor is specified twice in the DATASTORE_MAD arguments.

添加新的TM_MAD_CONF和DS_MAD_CONF部分:

TM_MAD_CONF = [
    NAME = "linstor", LN_TARGET = "NONE", CLONE_TARGET = "SELF", SHARED = "yes", ALLOW_ORPHANS="yes",
    TM_MAD_SYSTEM = "ssh,shared", LN_TARGET_SSH = "NONE", CLONE_TARGET_SSH = "SELF", DISK_TYPE_SSH = "BLOCK",
    LN_TARGET_SHARED = "NONE", CLONE_TARGET_SHARED = "SELF", DISK_TYPE_SHARED = "BLOCK"
]
DS_MAD_CONF = [
    NAME = "linstor", REQUIRED_ATTRS = "BRIDGE_LIST", PERSISTENT_ONLY = "NO",
    MARKETPLACE_ACTIONS = "export"
]

After making these changes, restart the OpenNebula service.

7.4.2. 配置节点

The Front-End node issues commands to the Storage and Host nodes through LINSTOR.

存储节点在本地保存vm的磁盘映像。

Host nodes are responsible for running instantiated VMs and typically have the storage for the images they need attached across the network through LINSTOR diskless mode.

All nodes must have DRBD9 and LINSTOR installed. This process is detailed in the User’s Guide for DRBD9

It is possible to have Front-End and Host nodes act as storage nodes in addition to their primary role provided that they the meet all the requirements for both roles.

Configuring the Front-end Node

Please verify that the control node(s) that you hope to communicate with are reachable from the front-end node. linstor node list for locally running LINSTOR controllers and linstor --controllers "<IP:PORT>" node list for remotely running LINSTOR Controllers is a handy way to test this.

Configuring Host Nodes

Host nodes must have LINSTOR satellite processes running on them and be members of the same LINSTOR cluster as the Front-End and Storage nodes, and may optionally have storage locally. If the oneadmin user is able to passwordlessly SSH between hosts then live migration may be used with the even with the SSH system datastore.

Configuring Storage Nodes

Only the front-end and host nodes require OpenNebula to be installed, but the oneadmin user must be able to passwordlessly access storage nodes. Refer to the OpenNebula installation guide for your distribution on how to manually configure the oneadmin user account.

The storage nodes must use storage pools created with a driver that’s capable of making snapshots, such as the thin LVM plug-in.

In this example preparation of thin-provisioned storage using LVM for LINSTOR, you must create a volume group and thinLV using LVM on each storage node.

The following is an example of this process using two physical volumes (/dev/sdX and /dev/sdY) and generic names for the volume group and thinpool. Take care to set the thinLV’s metadata volume to a reasonable size. once it becomes full it can be difficult to resize.

pvcreate /dev/sdX /dev/sdY
vgcreate drbdpool /dev/sdX /dev/sdY
lvcreate -l 95%VG --poolmetadatasize 8g -T /dev/drbdpool/drbdthinpool

Then you’ll create storage pool(s) on LINSTOR using this as the backing storage.

If you are using ZFS storage pools or thick-LVM, please use LINSTOR_CLONE_MODE copy otherwise you will have problems deleting linstor resources, because of ZFS parent-child snapshot relationships.

7.4.3. Permissions for the Administrative Account

The oneadmin, “Cloud Administrator”, user account must have passwordless sudo access to the mkfs command on the Storage nodes

oneadmin ALL=(root) NOPASSWD: /sbin/mkfs

Be sure to consider the groups that oneadmin should be added to to gain access to the devices and programs needed to access storage and instantiate VMs. For this add-on, the oneadmin user must belong to the disk group on all nodes to access the DRBD devices where images are held.

usermod -a -G disk oneadmin

7.4.4. Creating a New LINSTOR Datastore

Create a datastore configuration file named ds.conf and use the onedatastore tool to create a new datastore based on that configuration. There are two mutually exclusive deployment options: LINSTOR_AUTO_PLACE and LINSTOR_DEPLOYMENT_NODES. If both are configured, LINSTOR_AUTO_PLACE is ignored. For both of these options, BRIDGE_LIST must be a space separated list of all storage nodes in the LINSTOR cluster.

7.4.5. Creating Resource Groups

因为1.0.0版LINSTOR支持资源组。资源组是所有链接到该资源组的资源共享的设置的集中点。

Create a resource group and volume group for your datastore, it is mandatory to specify a storage-pool within the resource group, otherwise monitoring space for OpenNebula will not work. Here we create one with 2 node redundancy and use a created opennebula-storagepool:

linstor resource-group create OneRscGrp --place-count 2 --storage-pool opennebula-storagepool
linstor volume-group create OneRscGrp

Now add a OpenNebula datastore using the LINSTOR plug-in:

cat >ds.conf <<EOI
NAME = linstor_datastore
DS_MAD = linstor
TM_MAD = linstor
TYPE = IMAGE_DS
DISK_TYPE = BLOCK
LINSTOR_RESOURCE_GROUP = "OneRscGrp"
COMPATIBLE_SYS_DS = 0
BRIDGE_LIST = "alice bob charlie"  #node names
EOI

onedatastore create ds.conf

7.4.6. Plug-in Attributes

LINSTOR_CONTROLLERS

LINSTOR_CONTROLLERS can be used to pass a comma separated list of controller ips and ports to the LINSTOR client in the case where a LINSTOR controller process is not running locally on the Front-End, e.g.:

LINSTOR_CONTROLLERS = "192.168.1.10:8080,192.168.1.11:6000"

LINSTOR_RESOURCE_GROUP

LINSTOR_RESOURCE_GROUP attribute is used to associate an OpenNebula datastore with a LINSTOR resource group.

7.4.7. Deprecated Attributes

The following attributes are deprecated and were removed in version 2.0.

LINSTOR_CLONE_MODE

LINSTOR now automatically decides which clone mode it should use.

LINSTOR supports two different clone modes: snapshot and copy. These modes are set through the LINSTOR_CLONE_MODE attribute.

The default mode is snapshot. It uses a linstor snapshot and restores a new resource from this snapshot, which is then a clone of the image. This mode is usually faster than using the copy mode as snapshots are cheap copies.

The second mode is copy. It creates a new resource with the same size as the original and copies the data with dd to the new resource. This mode will be slower than snapshot, but is more robust as it doesn’t rely on any snapshot mechanism. It is also used if you are cloning an image into a different LINSTOR datastore.

LINSTOR_STORAGE_POOL

LINSTOR_STORAGE_POOL attribute is used to select the LINSTOR storage pool your datastore should use. If resource groups are used this attribute isn’t needed as the storage pool can be select by the auto select filter options.

If LINSTOR_AUTO_PLACE or LINSTOR_DEPLOYMENT_NODES is used and LINSTOR_STORAGE_POOL is not set, it will fallback to the DfltStorPool in LINSTOR.

LINSTOR_AUTO_PLACE

LINSTOR_AUTO_PLACE 选项采用一个冗余级别,即一个与存储节点总数之间的数字。资源根据冗余级别自动分配给存储节点。

LINSTOR_DEPLOYMENT_NODES

Using LINSTOR_DEPLOYMENT_NODES allows you to select a group of nodes that resources will always be assigned to. Please note that the bridge list still contains all of the storage nodes in the LINSTOR cluster.

7.4.8. Configuring LINSTOR as a System Datastore

LINSTOR driver can also be used as a system datastore, configuration is pretty similar to normal datastores, with a few changes:

cat >system_ds.conf <<EOI
NAME = linstor_system_datastore
TM_MAD = linstor
TYPE = SYSTEM_DS
LINSTOR_RESOURCE_GROUP = "OneSysRscGrp"
BRIDGE_LIST = "alice bob charlie"  # node names
EOI

onedatastore create system_ds.conf

还要将新的sys datastore id添加到镜像数据存储的 COMPATIBLE_SYS_DS (以逗号分隔),否则调度程序将忽略它们。

如果要使用易失性磁盘进行实时迁移,则需要为KVM启用 --unsafe 选项,请参见: opennebula doc

7.5. Live Migration

Live migration is supported even with the use of the SSH system datastore, as well as the nfs shared system datastore.

7.6. 可用空间报告

空闲空间的计算方式不同,这取决于资源是自动部署还是按节点部署。

对于每个节点放置的数据存储,将根据部署资源的所有节点中限制最严格的存储池报告可用空间。例如,总存储空间最小的节点的容量用于确定数据存储的总大小,而可用空间最小的节点用于确定数据存储中的剩余空间。

对于使用自动放置的数据存储,大小和剩余空间是根据LINSTOR报告的数据存储所使用的聚合存储池确定的。

8. LINSTOR Volumes in OpenStack

This chapter describes using LINSTOR to provision persistent, replicated, and high-performance block storage for OpenStack.

8.1. Introduction to OpenStack

OpenStack consists of a wide range of individual services; The service responsible for provisioning and managing block storage is called Cinder. Other openstack services such as the compute instance service Nova can request volumes from Cinder. Cinder will then make a volume accessible to the requesting service.

LINSTOR can integrate with Cinder using a volume driver. The volume driver translates calls to the Cinder API to LINSTOR commands. For example: requesting a volume from Cinder will create new resources in LINSTOR, Cinder Volume snapshots translate to snapshots in LINSTOR and so on.

8.2. Installing LINSTOR for OpenStack

An initial installation and configuration of DRBD and LINSTOR must be completed prior to using the OpenStack driver.

At this point you should be able to list your storage cluster nodes using the LINSTOR client:

$ linstor node info
╭────────────────────────────────────────────────────────────────────────────╮
┊ Node                      ┊ NodeType  ┊ Addresses                 ┊ State  ┊
╞════════════════════════════════════════════════════════════════════════════╡
┊ cinder-01.openstack.test  ┊ COMBINED  ┊ 10.43.224.21:3366 (PLAIN) ┊ Online ┊
┊ cinder-02.openstack.test  ┊ COMBINED  ┊ 10.43.224.22:3366 (PLAIN) ┊ Online ┊
┊ storage-01.openstack.test ┊ SATELLITE ┊ 10.43.224.11:3366 (PLAIN) ┊ Online ┊
┊ storage-02.openstack.test ┊ SATELLITE ┊ 10.43.224.12:3366 (PLAIN) ┊ Online ┊
┊ storage-03.openstack.test ┊ SATELLITE ┊ 10.43.224.13:3366 (PLAIN) ┊ Online ┊
╰────────────────────────────────────────────────────────────────────────────╯

You should configure one or more storage pools per node. This guide assumes the storage pool is named cinderpool. LINSTOR should list the storage pool for each node, including the diskless storage pool created by default.

$ linstor storage-pool list
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node                      ┊ Driver   ┊ PoolName        ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ cinder-01.openstack.test  ┊ DISKLESS ┊                 ┊              ┊               ┊ False        ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ cinder-02.openstack.test  ┊ DISKLESS ┊                 ┊              ┊               ┊ False        ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ storage-01.openstack.test ┊ DISKLESS ┊                 ┊              ┊               ┊ False        ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ storage-02.openstack.test ┊ DISKLESS ┊                 ┊              ┊               ┊ False        ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ storage-03.openstack.test ┊ DISKLESS ┊                 ┊              ┊               ┊ False        ┊ Ok    ┊
┊ cinderpool           ┊ storage-01.openstack.test ┊ LVM_THIN ┊ ssds/cinderpool ┊      100 GiB ┊       100 GiB ┊ True         ┊ Ok    ┊
┊ cinderpool           ┊ storage-02.openstack.test ┊ LVM_THIN ┊ ssds/cinderpool ┊      100 GiB ┊       100 GiB ┊ True         ┊ Ok    ┊
┊ cinderpool           ┊ storage-03.openstack.test ┊ LVM_THIN ┊ ssds/cinderpool ┊      100 GiB ┊       100 GiB ┊ True         ┊ Ok    ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

8.2.1. Upgrading the LINSTOR Driver

If this is a fresh installation, skip this section and continue with Installing the LINSTOR Driver.

Upgrading From 1.x to 2.x

Driver version 2 dropped some static configuration options in favour of managing these options at runtime using volume types.

Option in cinder.conf Status Replace with

linstor_autoplace_count

removed

Use the linstor:redundancy property on the volume type. Using a value of 0 for full cluster replication is not supported, use the advanced options of the LINSTOR autoplacer

linstor_controller_diskless

removed

No replacement needed, the driver will create a diskless resource on the cinder host when required

linstor_default_blocksize

removed

This setting had no effect.

linstor_default_storage_pool_name

deprecated

This setting is deprecated for removal in a future version. Use the linstor:storage_pool property on the volume type instead.

linstor_default_uri

deprecated

Replaced by the more aptly named linstor_uris.

linstor_default_volume_group_name

removed

Creating nodes and storage pools was completely removed in Driver version 2. See Installing LINSTOR for OpenStack

linstor_volume_downsize_factor

removed

This setting served no purpose, it was removed without replacement.

8.2.2. Installing the LINSTOR Driver

Starting with OpenStack Stein, the LINSTOR driver is part of the Cinder project. While the driver can be used as is, it might be missing features or fixes available in newer version. Due to OpenStacks update policy for stable versions, most improvements to the driver will not get back-ported to older stable releases.

LINBIT maintains a fork of the Cinder repository with all improvements to the LINSTOR driver backported to the supported stable versions. Currently, these are:

OpenStack Release Included Version LINBIT Version LINBIT Branch

yoga

1.0.1

2.0.0

linstor/stable/yoga

xena

1.0.1

2.0.0

linstor/stable/xena

wallaby

1.0.1

2.0.0

linstor/stable/wallaby

victoria

1.0.1

2.0.0

linstor/stable/victoria

ussuri

1.0.1

2.0.0

linstor/stable/ussuri

train

1.0.0

2.0.0

linstor/stable/train

stein

1.0.0

2.0.0

linstor/stable/stein

The exact steps to enable the LINSTOR Driver depend on your OpenStack distribution. In general, the python-linstor package needs to be installed on all hosts running the Cinder volume service. The next section will cover the installation process for common OpenStack distributions.

Installing on DevStack

DevStack is a great way to try out OpenStack in a lab environment. To use the most recent driver use the following DevStack configuration:

Listing 24. local.conf
# This ensures the LINSTOR Driver has access to the 'python-linstor' package.
#
# This is needed even if using the included driver!
USE_VENV=True
ADDITIONAL_VENV_PACKAGES=python-linstor

# This is required to select the LINBIT version of the driver
CINDER_REPO=https://github.com/LINBIT/openstack-cinder.git
# Replace linstor/stable/victoria with the reference matching your OpenStack release.
CINDER_BRANCH=linstor/stable/victoria
Installing on Kolla

Kolla packages OpenStack components in containers. They can then be deployed, for example using Kolla Ansible You can take advantage of the available customisation options for kolla containers to set up the LINSTOR driver.

To ensure that the required python-linstor package is installed, use the following override file:

Listing 25. template-override.j2
{% extends parent_template %}

# Cinder
{% set cinder_base_pip_packages_append = ['python-linstor'] %}

To install the LINBIT version of the driver, update your kolla-build.conf

Listing 26. /etc/kolla/kolla-build.conf
[cinder-base]
type = git
location = https://github.com/LINBIT/openstack-cinder.git
# Replace linstor/stable/victoria with the reference matching your OpenStack release.
reference = linstor/stable/victoria

To rebuild the Cinder containers, run:

# A private registry used to store the kolla container images
REGISTRY=deployment-registry.example.com
# The image namespace in the registry
NAMESPACE=kolla
# The tag to apply to all images. Use the release name for compatibility with kolla-ansible
TAG=victoria
kolla-build -t source --template-override template-override.j2 cinder --registry $REGISTRY --namespace $NAMESPACE --tag $TAG
Kolla Ansible Deployment

When deploying OpenStack using Kolla Ansible, you need to verify that:

  • the custom Cinder images, created in the section above, are used deployment of Cinder services is enabled.

Listing 27. /etc/kolla/globals.yml
# use "source" images
kolla_install_type: source
# use the same registry as for running kolla-build above
docker_registry: deployment-registry.example.com
# use the same namespace as for running kolla-build above
docker_namespace: kolla
# deploy cinder block storage service
enable_cinder: "yes"
# disable verification of cinder back ends, kolla-ansible only supports a small subset of available back ends for this
skip_cinder_backend_check: True
# add the LINSTOR back end to the enabled back ends. For back end configuration see below
cinder_enabled_backends:
  - name: linstor-drbd

You can place the LINSTOR driver configuration in one of the override directories for kolla-ansible. For more details on the available configuration options, see the section below.

Listing 28. /etc/kolla/config/cinder/cinder-volume.conf
[linstor-drbd]
volume_backend_name = linstor-drbd
volume_driver = cinder.volume.drivers.linstordrv.LinstorDrbdDriver
linstor_uris = linstor://cinder-01.openstack.test,linstor://cinder-02.openstack.test
OpenStack Ansible Deployment

OpenStack Ansible provides Ansible playbooks to configure and deploy of OpenStack environments. It allows for fine-grained customization of the deployment, letting you set up the LINSTOR driver directly.

Listing 29. /etc/openstack_ansile/user_variables.yml
cinder_git_repo: https://github.com/LINBIT/openstack-cinder.git
cinder_git_install_branch: linstor/stable/victoria

cinder_user_pip_packages:
  - python-linstor

cinder_backends: (1)
  linstor-drbd:
   volume_backend_name: linstor-drbd
   volume_driver: cinder.volume.drivers.linstordrv.LinstorDrbdDriver
   linstor_uris: linstor://cinder-01.openstack.test,linstor://cinder-02.openstack.test
1 A detailed description of the available back end parameters can be found in the section below.
Generic Cinder Deployment

For other forms of OpenStack deployments, this guide can only provide non-specific hints.

To update the LINSTOR driver version, find your Cinder installation. Some likely paths are:

/usr/lib/python*/dist-packages/cinder/
/usr/lib/python*/site-packages/cinder/

The LINSTOR driver consists of a single file called linstordrv.py, located in the Cinder directory:

$CINDER_PATH/volume/drivers/linstordrv.py

To update the driver, replace the file with one from the LINBIT repository

RELEASE=linstor/stable/victoria
curl -fL "https://raw.githubusercontent.com/LINBIT/openstack-cinder/$RELEASE/cinder/volume/drivers/linstordrv.py" > $CINDER_PATH/volume/drivers/linstordrv.py

You might also need to remove the Python cache for the update to be registered:

rm -rf $CINDER_PATH/volume/drivers/__pycache__

8.3. Configuring a LINSTOR Back End for Cinder

To use the LINSTOR driver, configure the Cinder volume service. This is done by editing the Cinder configuration file and then restarting the Cinder Volume service.

Most of the time, the Cinder configuration file is located at /etc/cinder/cinder.conf. Some deployment options allow manipulating this file in advance. See the section above for specifics.

To configure a new volume back end using LINSTOR, add the following section to cinder.conf

[linstor-drbd]
volume_backend_name = linstor-drbd (1)
volume_driver = cinder.volume.drivers.linstordrv.LinstorDrbdDriver (2)
linstor_uris = linstor://cinder-01.openstack.test,linstor://cinder-02.openstack.test (3)
linstor_trusted_ca = /path/to/trusted/ca.cert (4)
linstor_client_key = /path/to/client.key (5)
linstor_client_cert = /path/to/client.cert (5)
# Deprecated or removed in 2.0.0
linstor_default_storage_pool_name = cinderpool (6)
linstor_autoplace_count = 2 (7)
linstor_controller_diskless = true (8)
# non-linstor-specific options
... (9)
The parameters described here are based on the latest release provided by LINBIT. The driver included in OpenStack might not support all of these parameters. Consult the OpenStack driver documentation to learn more.
1 The name of the volume back end. Needs to be unique in the Cinder configuration. The whole section should share the same name. This name is referenced again in cinder.conf in the enabled_backends setting and when creating a new volume type.
2 The version of the LINSTOR driver to use. There are two options:
  • cinder.volume.drivers.linstordrv.LinstorDrbdDriver

  • cinder.volume.drivers.linstordrv.LinstorIscsiDriver

    Which driver you should use depends on your LINSTOR set up and requirements. Details on each choice are documented in the section below.

3 The URL(s) of the LINSTOR Controller(s). Multiple Controllers can be specified to make use of Linstor High Availability. If not set, defaults to linstor://localhost.
In driver versions prior to 2.0.0, this option is called linstor_default_uri
4 If HTTPS is enabled the referenced certificate is used to verify the LINSTOR Controller authenticity.
5 If HTTPS is enabled the referenced key and certificate will be presented to the LINSTOR Controller for authentication.
6 Deprecated in 2.0.0, use volume types instead. The storage pools to use when placing resources. Applies to all diskfull resources created. Defaults to DfltStorPool.
7 Removed in 2.0.0, use volume types instead. The number of replicas to create for the given volume. A value of 0 will create a replica on all nodes. Defaults to 0.
8 Removed in 2.0.0, volumes are created on demand by the driver. If set to true, ensures that at least one (diskless) replica is deployed on the Cinder Controller host. This is useful for ISCSI transports. Defaults to true.
9 You can specify more generic Cinder options here, for example target_helper = tgtadm for the ISCSI connector.
You can also configure multiple LINSTOR back ends, choosing a different name and configuration options for each.

After configuring the LINSTOR back end, it should also be enabled. Add it to the list of enabled back ends in cinder.conf, and optionally set is as the default back end:

[DEFAULT]
...
default_volume_type = linstor-drbd-volume
enabled_backends = lvm,linstor-drbd
...

As a last step, if you changed the Cinder configuration or updated the driver itself, you need to restart the Cinder service(s). Please check the documentation for your OpenStack Distribution on how to restart services.

8.3.1. Choosing a Transport Protocol

The Transport Protocol in Cinder is how clients (for example nova-compute) access the actual volumes. With LINSTOR, you can choose between two different drivers that use different transports.

  • cinder.volume.drivers.linstordrv.LinstorDrbdDriver, which uses DRBD as transport

  • cinder.volume.drivers.linstordrv.LinstorIscsiDriver, which uses ISCSI as transport

Using DRBD as the Transport Protocol

The LinstorDrbdDriver works by ensuring a replica of the volume is available locally on the node where a client (that is, nova-compute) issued a request. This only works if all compute nodes are also running LINSTOR Satellites that are part of the same LINSTOR cluster.

The advantages of this option are:

  • Once set up, the Cinder host is no longer involved in the data path. All read and write to the volume are handled by the local DRBD module, which will handle replication across its configured peers.

  • Since the Cinder host is not involved in the data path, any disruptions to the Cinder service do not affect volumes that are already attached.

Known limitations:

  • Not all hosts and hypervisors support using DRBD volumes. This restricts deployment to Linux hosts and kvm hypervisors.

  • Resizing of attached and in-use volumes does not fully work. While the resize itself is successful, the compute service will not propagate it to the VM until after a restart.

  • Multi-attach (attaching the same volume on multiple VMs) is not supported.

  • Encrypted volumes only work if udev rules for DRBD devices are in place.

    udev rules are either part of the drbd-utils package or have their own drbd-udev package.
Using iSCSI as the Transport Protocol

The default way to export Cinder volumes is through iSCSI. This has the advantage of maximum compatibility as iSCSI can be used with every hypervisor, be it VMWare, Xen, HyperV, or KVM.

缺点是,所有数据都必须发送到Cinder节点,由(userspace)iSCSI守护进程处理;这意味着数据需要通过内核/userspace边界,这些转换将消耗一些性能。

Another drawback is the introduction of a single point of failure. If a Cinder node running the iSCSI daemon crashes, other nodes lose access to their volumes. There are ways to configure Cinder for automatic fail-over to mitigate this, but it requires considerable effort.

In driver versions prior to 2.0.0, the Cinder host needs access to a local replica of every volume. This can be achieved by either setting linstor_controller_diskless=True or using linstor_autoplace_count=0. Newer driver versions will create such a volume on demand.

8.3.2. Verifying the Status of LINSTOR Back Ends

To verify that all back ends are up and running, you can use the OpenStack command line client:

$ openstack volume service list
+------------------+----------------------------------------+------+---------+-------+----------------------------+
| Binary           | Host                                   | Zone | Status  | State | Updated At                 |
+------------------+----------------------------------------+------+---------+-------+----------------------------+
| cinder-scheduler | cinder-01.openstack.test               | nova | enabled | up    | 2021-03-10T12:24:37.000000 |
| cinder-volume    | [email protected]  | nova | enabled | up    | 2021-03-10T12:24:34.000000 |
| cinder-volume    | [email protected] | nova | enabled | up    | 2021-03-10T12:24:35.000000 |
+------------------+----------------------------------------+------+---------+-------+----------------------------+

If you have the Horizon GUI deployed, check Admin > System Information > Block Storage Service instead.

In the above example all configured services are enabled and up. If there are any issues, please check the logs of the Cinder Volume service.

8.4. Creating a New Volume Type for LINSTOR

Before creating volumes using Cinder, you have to create a volume type. This can be done using the command line:

# Create a volume using the default back end
$ openstack volume type create default
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| description | None                                 |
| id          | 58365ffb-959a-4d91-8821-5d72e5c39c26 |
| is_public   | True                                 |
| name        | default                              |
+-------------+--------------------------------------+
# Create a volume using a specific back end
$ openstack volume type create --property volume_backend_name=linstor-drbd linstor-drbd-volume
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| description | None                                 |
| id          | 08562ea8-e90b-4f95-87c8-821ac64630a5 |
| is_public   | True                                 |
| name        | linstor-drbd-volume                  |
| properties  | volume_backend_name='linstor-drbd'   |
+-------------+--------------------------------------+

Alternatively, you can create volume types using the Horizon GUI. Navigate to Admin > Volume > Volume Types and click “Create Volume Type”. You can assign it a back end by adding the volume_backend_name as “Extra Specs” to it.

8.4.1. Advanced Configuration of Volume Types

Each volume type can be customized by adding properties or “Extra Specs” as they are called in the Horizon GUI.

To add a property to a volume type on the command line use:

openstack volume type set linstor_drbd_b --property linstor:redundancy=5

Alternatively, you can set the property using the GUI by navigating tp Admin > Volume > Volume Types. In the Actions column, open the dropdown menu and click the View Extra Specs button. This opens a dialog you can use to create, edit and delete properties.

Available Volume Type Properties
linstor:diskless_on_remaining

Create diskless replicas on non-selected nodes after auto-placing.

linstor:do_not_place_with_regex

Do not place the resource on a node which has a resource with a name matching the regex.

linstor:layer_list

Comma-separated list of layers to apply for resources. If empty, defaults to DRBD,Storage.

linstor:provider_list

Comma-separated list of providers to use. If empty, LINSTOR will automatically choose a suitable provider.

linstor:redundancy

Number of replicas to create. Defaults to two.

linstor:replicas_on_different

A comma-separated list of key or key=value items used as autoplacement selection labels when autoplace is used to determine where to provision storage.

linstor:replicas_on_same

A comma-separated list of key or key=value items used as autoplacement selection labels when autoplace is used to determine where to provision storage.

linstor:storage_pool

Comma-separated list of storage pools to use when auto-placing.

linstor:property:<key>

If a <key> is prefixed by linstor:property:, it is interpreted as a LINSTOR property. The property gets set on the Resource Group created for the volume type.

OpenStack does not allow for / in property names. If a LINSTOR property name contains a / replace it with a :.

For example: To change the quorum policy, DrbdOptions/auto-quorum needs to be set. This can be done by setting the linstor:property:DrbdOptions:auto-quorum property in OpenStack.

8.5. Using Volumes

Once you have a volume type configured, you can start using it to provision new volumes.

For example, to create a simple 1Gb volume on the command line you can use:

openstack volume create --type linstor-drbd-volume --size 1 \
  --availability-zone nova linstor-test-vol
openstack volume list
If you set default_volume_type = linstor-drbd-volume in your /etc/cinder/cinder.conf, you may omit the --type linstor-drbd-volume from the openstack volume create …​ command above.

8.6. Troubleshooting

This section describes what to do in case you encounter problems with using LINSTOR volumes and snapshots.

8.6.1. Checking for Error Messages in Horizon

Every volume and snapshot has a Messages tab in the Horizon dashboard. In case of errors, the list of messages can be used as a starting point for further investigation. Some common messages in case of errors:

create volume from backend storage:Driver failed to create the volume.

There was an error creating a new volume. Check the Cinder Volume service logs for more details

schedule allocate volume:Could not find any available weighted backend.

If this is the only error message, this means the Cinder Scheduler could not find a volume back end suitable for creating the volume. This is most likely because:

  • The volume back end is offline. See Verifying the Status of LINSTOR Back Ends

  • The volume back end has not enough free capacity to fulfil the request. Check the output of cinder get-pools --detail and linstor storage-pool list to ensure that the requested capacity is available.

8.6.2. Checking the Cinder Volume Service

The LINSTOR driver is called as part of the Cinder Volume service.

Distribution Log location or command

DevStack

journalctl -u [email protected]

8.6.3. Checking the Compute Service Logs

Some issues will not be logged in the Cinder Service but in the actual consumer of the volumes, most likely the compute service (Nova). As with the volume service, the exact host and location to check depends on your OpenStack distribution:

Distribution Log location or command

DevStack

journalctl -u [email protected]

9. Docker中的LINSTOR卷

本章介绍docker中由https://github.com/LINBIT/linstor-docker-volume-go[LINSTOR Docker Volume Plugin]管理的LINSTOR卷。

9.1. Introduction to Docker

Docker 是一个以Linux容器的形式开发、发布和运行应用程序的平台。对于需要数据持久性的有状态应用程序,Docker支持使用持久 卷驱动程序

LINSTOR Docker Volume Plugin是一个卷驱动程序,它为docker容器提供来自linstor集群的持久卷。

9.2. Installing the LINSTOR Plug-in for Docker

To install the linstor-docker-volume plug-in provided by LINBIT, you’ll need to have a working LINSTOR cluster. After that the plug-in can be installed from the public docker hub.

# docker plugin install linbit/linstor-docker-volume --grant-all-permissions
The --grant-all-permissions flag will automatically grant all permissions needed to successfully install the plug-in. If you’d like to manually accept these, omit the flag from the command above.

The implicit :latest tag is the latest amd64 version. We currently also build for arm64 with the according tag. Installing the arm64 plugin looks like this:

# docker plugin install linbit/linstor-docker-volume:arm64 --grant-all-permissions

9.3. Configuring the LINSTOR Plug-in for Docker

As the plug-in has to communicate to the LINSTOR Controller software using the LINSTOR Python library, we must tell the plug-in where to find the LINSTOR controller node in its configuration file:

# cat /etc/linstor/docker-volume.conf
[global]
controllers = linstor://hostnameofcontroller

更广泛的例子如下:

# cat /etc/linstor/docker-volume.conf
[global]
storagepool = thin-lvm
fs = ext4
fsopts = -E discard
size = 100MB
replicas = 2

9.4. Using the LINSTOR Plug-in for Docker

The following are some examples of how you might use the LINSTOR Docker Volume Plug-in. In the following we expect a cluster consisting of three nodes (alpha, bravo, and charlie).

9.4.1. Typical Docker Pattern

在节点alpha上:

$ docker volume create -d linbit/linstor-docker-volume \
        --opt fs=xfs --opt size=200 lsvol
$ docker run -it --rm --name=cont \
        -v lsvol:/data --volume-driver=linbit/linstor-docker-volume busybox sh
$ [email protected]: echo "foo" > /data/test.txt
$ [email protected]: exit

在bravo节点上:

$ docker run -it --rm --name=cont \
        -v lsvol:/data --volume-driver=linbit/linstor-docker-volume busybox sh
$ [email protected]: cat /data/test.txt
  foo
$ [email protected]: exit
$ docker volume rm lsvol

9.4.2. One Diskful Assignment by Name, Two Nodes Diskless

$ docker volume create -d linbit/linstor-docker-volume --opt nodes=bravo lsvol

9.4.3. One Diskful Assignment, No Matter Where, Two Nodes Diskless

$ docker volume create -d linbit/linstor-docker-volume --opt replicas=1 lsvol

9.4.4. Two Diskful Assignments by Name, and One Diskless

$ docker volume create -d linbit/linstor-docker-volume --opt nodes=alpha,bravo lsvol

9.4.5. Two Diskful Assignments, No Matter Where, One Node Diskless

$ docker volume create -d linbit/linstor-docker-volume --opt replicas=2 lsvol

9.4.6. Using LINSTOR Volumes with Services from Docker Swarm Manager Node

$ docker service create \
        --mount type=volume,src=lsvol,dst=/data,volume-driver=linbit/linstor-docker-volume \
        --name swarmsrvc busybox sh -c "while true; do sleep 1000s; done"
Docker services do not accept the -v or --volume syntax, you must use the --mount syntax. Docker run will accept either syntax.

10. LINSTOR Gateway for Highly Available NFS/iSCSI Storage

LINSTOR Gateway manages highly available iSCSI targets and NFS exports by leveraging both LINSTOR and DRBD Reactor.

10.1. LINSTOR Gateway Requirements

LINSTOR Gateway requires an initialized LINSTOR cluster, DRBD Reactor, as well as NFS, iSCSI, or both utilities be installed and configured before it can be used. The following sections cover these requirements in detail.

10.1.1. LINSTOR Cluster

The LINSTOR cluster needs to be set up as a prerequisite to using LINSTOR Gateway. For more detail regarding LINSTOR’s configuration options, please refer to the LINSTOR User’s Guide.

For both iSCSI and NFS, a LINSTOR storage-pool , resource-group and volume-group for LINSTOR Gateway needs to be created before use. The following section provides example commands for setting up the prerequisites in a three-node LINSTOR cluster.

Create a LVM backed storage-pool on each node using the physical device /dev/sdb:

# linstor physical-storage create-device-pool
          --pool-name lvpool LVM LINSTOR1 /dev/sdb --storage-pool lvmpool
# linstor physical-storage create-device-pool
          --pool-name lvpool LVM LINSTOR2 /dev/sdb --storage-pool lvmpool
# linstor physical-storage create-device-pool
          --pool-name lvpool LVM LINSTOR3 /dev/sdb --storage-pool lvmpool

Create resource-groups and volume-groups backed by the storage-pool created in the previous command:

# linstor resource-group create iscsi_group --storage-pool lvmpool --place-count 2
# linstor resource-group create nfs_group --storage-pool lvmpool --place-count 3
# linstor volume-group create iscsi_group
# linstor volume-group create nfs_group

LINSTOR Gateway requires modification of the LINSTOR Satellite’s configuration on each Satellite node. Edit or create the following file using your preferred text editor:

# vim /etc/linstor/linstor_satellite.toml

Add the following content to the Satellite configuration:

[files]
  allowExtFiles = [
    "/etc/systemd/system",
    "/etc/systemd/system/linstor-satellite.service.d",
    "/etc/drbd-reactor.d"
  ]

Save the changes to the Satellite configuration and restart the satellite service on all nodes to load the changes.

# systemctl restart linstor-satellite

10.1.2. DRBD Reactor

DRBD Reactor is a daemon that will orchestrate the iSCSI, NFS or NVMe-oF resources in our cluster. It must be installed and the service must be enabled on all nodes in the cluster.

DRBD Reactor’s main function could be summarized as such: each DRBD Reactor daemon tries to take over DRBD Reactor managed services. The daemon that wins the race prevents other nodes from activating the services until the original winner is no longer able to. When that happens, a new winner will take over DRBD Reactor managed services, therefore achieving high-availability.

For details regarding DRBD Reactor’s installation and configuration options, refer to the DRBD Reactor GitHub page.

Start and enable the DRBD Reactor service on all nodes:

# systemctl enable --now drbd-reactor

In addition to this, DRBD Reactor needs to be configured to automatically reload when its configuration changes. Run this snippet on all nodes, as described in the DRBD Reactor README:

# cp examples/drbd-reactor-reload.{path,service} /etc/systemd/system/
# systemctl enable --now drbd-reactor-reload.path

DRBD Reactor uses Pacemaker’s resource agents when integrated with LINSTOR Gateway. You also need to install resource agents on all nodes in the cluster. This package is named resource-agents in both RPM and DEB based distributions:

# yum -y install resource-agents

10.1.3. iSCSI and NFS Utilities

LINSTOR Gateway requires the LIO iSCSI implementation be installed on all nodes.

The targetcli package can be installed on RPM based systems using the following command:

# yum -y install targetcli

For NFS support in LINSTOR Gateway, nfs-server utilities need to be installed on each node.

The nfs-utils package can be installed on RPM based systems using the following command:

# yum -y install nfs-utils

The NFS server should not be enabled in systemd since that will conflict with DRBD Reactors ability to manage the service. Disable the nfs-server service and check that it has been disabled using the following commands:

# systemctl disable nfs-server --now
# systemctl status nfs-server

Ensure that the output of the command above lists the service as inactive and disabled:

● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; ..snip..)
   Active: inactive (dead)

10.2. Verifying Requirements are Satisfied

The last thing we can do before starting to use LINSTOR Gateway is check that we’ve satisfied the prerequisites outlined in the previous sections.

10.2.1. Verifying Components are Installed

Let’s check that all the required components are present. This guide assumes you already installed and configured a LINSTOR cluster complete with storage-pools, resource-groups, and volume-groups before using linstor-gateway.

In addition to the initialized LINSTOR cluster, the following tools need to be present on all nodes:

  • linstor-client

  • drbd-reactor

  • targetcli

  • nfs-utils (RPM) or nfs-common (DEB)

  • nfs-server (RPM) or nfs-kernel-server (DEB)

  • resource-agents

LINSTOR Gateway provides a utility to check that the prerequisite tools are present:

# linstor-gateway check-health

This command should print something similar to the output below if you installed all of the required components. If an error is reported, you must resolve the error before proceeding.

[✓] LINSTOR
[✓] drbd-reactor
[✓] Resource Agents
[✓] iSCSI
[✓] NVMe-oF
[✓] NFS

10.2.2. Verifying LINSTOR Cluster Initialization

Verify that the LINSTOR cluster is initialized properly by comparing your outputs are similar to the outputs in the commands below.

Ensure all your LINSTOR nodes are listed as a Satellite or Combined type, and that you have 3 (or more) to support quorum:

# linstor node list
╭────────────────────────────────────────────────────────────╮
┊ Node     ┊ NodeType  ┊ Addresses                  ┊ State  ┊
╞════════════════════════════════════════════════════════════╡
┊ LINSTOR1 ┊ COMBINED  ┊ 172.16.16.111:3366 (PLAIN) ┊ Online ┊
┊ LINSTOR2 ┊ SATELLITE ┊ 172.16.16.112:3366 (PLAIN) ┊ Online ┊
┊ LINSTOR3 ┊ SATELLITE ┊ 172.16.16.113:3366 (PLAIN) ┊ Online ┊
╰────────────────────────────────────────────────────────────╯

Check that LINSTOR’s storage-pool list includes an LVM or ZFS backed storage-pool:

# linstor storage-pool list
╭─────────────────────────────────────────────────────────..snip..─────────╮
┊ StoragePool          ┊ Node     ┊ Driver   ┊ PoolName ┊ ..snip.. ┊ State ┊
╞═════════════════════════════════════════════════════════..snip..═════════╡
┊ DfltDisklessStorPool ┊ LINSTOR1 ┊ DISKLESS ┊          ┊ ..snip.. ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ LINSTOR2 ┊ DISKLESS ┊          ┊ ..snip.. ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ LINSTOR3 ┊ DISKLESS ┊          ┊ ..snip.. ┊ Ok    ┊
┊ lvmpool              ┊ LINSTOR1 ┊ LVM      ┊ lvpool   ┊ ..snip.. ┊ Ok    ┊
┊ lvmpool              ┊ LINSTOR2 ┊ LVM      ┊ lvpool   ┊ ..snip.. ┊ Ok    ┊
┊ lvmpool              ┊ LINSTOR3 ┊ LVM      ┊ lvpool   ┊ ..snip.. ┊ Ok    ┊
╰─────────────────────────────────────────────────────────..snip..─────────╯

Check that you’ve created at least one LINSTOR resource-group that uses your storage-pool. Also verify that each resource-group has a corresponding volume-group:

# linstor resource-group list
╭────────────────────────────────────────────────────────────────╮
┊ ResourceGroup ┊ SelectFilter            ┊ VlmNrs ┊ Description ┊
╞════════════════════════════════════════════════════════════════╡
┊ DfltRscGrp    ┊ PlaceCount: 2           ┊        ┊             ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ iscsi_group   ┊ PlaceCount: 2           ┊ 0      ┊             ┊
┊               ┊ StoragePool(s): lvmpool ┊        ┊             ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ nfs_group     ┊ PlaceCount: 3           ┊ 0      ┊             ┊
┊               ┊ StoragePool(s): lvmpool ┊        ┊             ┊
╰────────────────────────────────────────────────────────────────╯
# linstor volume-group list iscsi_group
╭──────────────────╮
┊ VolumeNr ┊ Flags ┊
╞══════════════════╡
┊ 0        ┊       ┊
╰──────────────────╯
# linstor volume-group list nfs_group
╭──────────────────╮
┊ VolumeNr ┊ Flags ┊
╞══════════════════╡
┊ 0        ┊       ┊
╰──────────────────╯

10.3. Creating iSCSI Targets

Once the preparations are complete, you can start creating iSCSI luns. The linstor-gateway command line utility will be used to manage all iSCSI related actions.

Use linstor-gateway iscsi help for detailed information regarding the iscsi subcommand.

The following command will create a new DRBD resource in the LINSTOR cluster with the specified name and resource-group. This command also creates the DRBD Reactor configuration files to enable high availability of the iSCSI target.

# linstor-gateway iscsi create iqn.2019-08.com.linbit:example 192.168.122.181/24 1G \
                        --username=foo --password=bar --resource-group=iscsi_group

After running the command above, you will have a 1GiB iSCSI target with CHAP authentication enabled using the username and password provided. It will be discoverable on the IP address provided in the command. The target will be backed by a DRBD device managed by LINSTOR. The DRBD resource was created by LINSTOR in the iscsi_group resource-group. The DRBD Reactor configuration files created by the above command can be found in /etc/drbd-reactor.d/.

You can list LINSTOR Gateway created iSCSI resources using the linstor-gateway iscsi list command:

# linstor-gateway iscsi list
+--------------------------------+--------------------+---------------+-----+---------------+
|              IQN               |     Service IP     | Service state | LUN | LINSTOR state |
+--------------------------------+--------------------+---------------+-----+---------------+
| iqn.2019-08.com.linbit:example | 192.168.122.181/24 | Started       |   1 | OK            |
+--------------------------------+--------------------+---------------+-----+---------------+
You can check the DRBD Reactor status using the drbd-reactorctl status command.

10.4. Deleting iSCSI Targets

The following command will delete the iSCSI target from DRBD Reactor as well as the LINSTOR cluster:

# linstor-gateway delete -i iqn.2021-04.com.linbit:lun4 -l 4

10.5. Creating NFS Exports

Before creating a NFS export you need to tell LINSTOR which filesystem the DRBD resource should be formatted with. This is done by setting the FileSystem/Type property on the resource-group created for NFS exports. Use the following LINSTOR command to do so:

# linstor resource-group set-property nfs_group FileSystem/Type ext4
You only need to set this once per resource-group, and only on the resource-group created specifically for LINSTOR Gateway’s NFS exports.

Finally, the following command will create a HA NFS export in the cluster. This single command will create a new resource within the LINSTOR cluster using the specified name and resource-group. This command also creates the DRBD Reactor configuration files to enable high availability of the NFS export.

# linstor-gateway nfs create nfstest 172.16.16.102/32 1G \
                  --allowed-ips=172.16.16.0/24 --resource-group=nfs_group

After running the command above, you will have a 1GiB NFS export with the specified allowed-ips able to mount the export using the IP address specified as the --service-ip. The exports will be backed by a DRBD device managed by LINSTOR. The DRBD resource was created by LINSTOR in the nfs_group resource-group. The DRBD Reactor configuration files created by the above command can be found in /etc/drbd-reactor.d/.

You can list LINSTOR Gateway created NFS resources using the linstor-gateway nfs list command:

# LINSTOR-gateway nfs list
+----------+------------------+---------------+------------------------------+---------------+
| Resource |    Service IP    | Service state |          NFS export          | LINSTOR state |
+----------+------------------+---------------+------------------------------+---------------+
| nfstest  | 172.16.16.102/32 | Started       | /srv/gateway-exports/nfstest | OK            |
+----------+------------------+---------------+------------------------------+---------------+
You can check the DRBD Reactor status using the drbd-reactorctl status command.

10.6. Deleting NFS Exports

The following command will delete the NFS export from DRBD Reactor as well as the LINSTOR cluster:

# linstor-gateway nfs delete -r nfstest

11. LINSTOR Exos Integration

The Exos storage manager from Seagate could be configured as one large block device managed by LINSTOR like a local drive, but this would prevent concurrent sharing of LINSTOR resources between multiple servers out of the same pool.

LINSTOR integration with Exos enables multiple server nodes to allocate and connect to LINSTOR resources serviced by the same Exos pool. Therefore all of the Exos storage management features such as SSD/HDD tiering, SSD caching, snapshots, and thin provisioning are available for LINSTOR resources and Kubernetes Storage Classes.

After configuration, LINSTOR will dynamically map Resource replicas as LUNs presented to server nodes through one of the two Exos controllers.

Since the Exos controllers are managed by a secure network API, LINSTOR must be configured with proper networking and username/password combination. The diagram below is showing the relationship between LINSTOR cluster and Exos Enclosures.

ExosIntegeration
Multi-host setup allows up to eight LINSTOR nodes to be directly connected with 48Gbit SAS links for low latency and high throughput.

Load balancing and server failover are managed & enabled by LINSTOR while volume creation is handled by the Exos hardware RAID engine.

The Exos storage provider in LINSTOR offers native integration with Exos’ REST-API.

This section will describe how to enable Exos integration and configure LINSTOR to manage storage backed by an Exos enclosure.

Exos storage systems offer a feature rich set of configuration options to match any enterprise storage demand. To maximize ease of use, this guide is based on the following defaults and assumptions:

  1. Dual Controllers – Exos systems controllers are Active/Active with automatic failover. Both controllers IP address must be configured also in the LINSTOR properties for full support.

  2. Dual Exos Pools – Optimal performance is achieved when data from pool A is accessed through Controller A. If a node is connected to both Controller A and B of same controller, LINSTOR will configure Linux multipath which will detect best route.

  3. Exos Pool Serial Numbers – When a Exos pool is created, it receives a unique serial number. Each one has to be configured as a backing storage in LINSTOR to create a link between Exos enclosure & LINSTOR. With that information, LINSTOR can understand if you are referring to Exos Pool A or Pool B.

  4. Creating Exos Pools – The administrator is required to create Exos Pools A and B prior to configuring LINSTOR. Exos features such as thin provisioning, auto tiering, and snapshot options are selected at this time.

  5. Replicas Within Enclosures – Exos system have redundant controllers, power supplies and communication paths to the drives. Some administrators may require that resource replicas are not stored in the same enclosure. In this case the administrator must create multiple LINSTOR Pools configured with only one Exos pool member from each enclosure.

11.1. Exos Properties as a LINSTOR Storage Provider

LINSTOR’s native integration with Exos is configured by setting a few properties on the LINSTOR Controller and creating the appropriate LINSTOR objects specific to your Exos enclosures, as described in the sections below.

The information in the table below is needed from your Exos enclosures. This information will be used to populate the appropriate LINSTOR Controller properties and LINSTOR objects in the sub-sections that follow.

Table 2. Required Information
Exos Information Description Placeholder in Command Examples

Exos Enclosure Name

Uniquely selected by the Admin for a given Exos enclosure

<exos_encl_name>

Controller Hostname

The DNS resolvable hostname for one of the Controllers

<exos_ctrlr_name>

Controller IP

IP address of controller

<exos_ctrlr_ip>

REST-API Username

Username for REST-API of all Exos controllers under the given enclosure

<exos_rest_user>

REST-API Password

Password for REST-API of all Exos controllers under the given enclosure

<exos_rest_pass>

Exos Pool Serial Number

The serial number of an Exos pool to become a member of a LINSTOR Pool

<exos_pool_sn>

11.2. Configuring a LINSTOR and Exos Integration

Configuring a topology of LINSTOR server nodes and multiple Exos Storage systems is described by these steps:

  1. Setting global or unique Exos Controller usernames and passwords.

  2. Defining Exos enclosures and Controller network identities.

  3. Creating node to enclosure to pool mapping matching physical SAS cabling.

11.2.1. Setting Exos Usernames and Passwords

Usernames and passwords can be unique for each Exos enclosure or be common for all enclosures depending on how the system administrator has deployed the Exos systems. The default Exos username and password will be used if not set for a given Exos controller.

The defaults are set as follows:

# linstor exos set-defaults --username <exos_rest_name>
# linstor exos set-defaults --password <exos_rest_pass>

Unique usernames and passwords For Exos controllers are set by:

# linstor controller set-property
    StorDriver/Exos/<exos_encl_name>/username <exos_rest_name>
# linstor controller set-property
    StorDriver/Exos/<exos_encl_name>/Password <exos_rest_pass>
Passwords entered in this fashion will show up as plain text when using get-defaults.

With the above command, LINSTOR will store your password in plain text in the LINSTOR properties and visible by a simple linstor controller list-properties command. You can hide it under an environment variable, and use the UsernameEnv and/or PasswordEnv properties. This tells LINSTOR to look in environment variable for the actual username/password, as shown in the following example:

LINSTOR will not modify the environment variables, only read from them. Storage admin has to verify that the environment variables are correctly set.
# echo $EXOS_PW
mySecretPassword
# linstor controller set-property \
    StorDriver/Exos/<exos_encl_name>/PasswordEnv EXOS_PW

If both property-versions (i.e. Password and PasswordEnv) are set, the non-environment version is preferred.

If the satellite is started before the environment variable is set, the satellite needs to be restarted to see the new environment variable.

11.2.2. Defining Exos Enclosures and Controller Identities

Registering an Exos enclosure in LINSTOR can be done with the create command:

# linstor exos create <exos_encl_name> <exos_ctrl_a_ip> [<exos_ctrl_b_ip>]

If no special --username or --password is given, the above mentioned defaults are used.

The Controller’s DNS name and IP address may be used interchangeably.

If you want to use a hostname that is not DNS resolvable to reference your Exos enclosure within LINSTOR, you may use any name in place of <exos_hostname>, but you will also have to supply the enclosure’s IP address: linstor node create <desired_name> <enclosure_ip>

Use the following example to create and inspect the current controller settings:

|==================================================================|
# linstor exos create Alpha 172.16.16.12 172.16.16.13
# linstor exos list
+------------------------------------------------------------------+
| Enclosure | Ctrl A IP    | Ctrl B IP    | Health | Health Reason |
| Alpha     | 172.16.16.12 | 172.16.16.13 | OK     |               |
+------------------------------------------------------------------+

For a more in-depth view, you can always ask the LINSTOR controller or the LINSTOR nodes for the Exos-related properties:

# linstor controller list-properties | grep Exos
| StorDriver/Exos/Alpha/A/IP                | 172.16.16.12         |
| StorDriver/Exos/Alpha/B/IP                | 172.16.16.13         |

11.2.3. Creating Node to Enclosure to Pool Mapping

A LINSTOR Satellite node can be created as usual.

# linstor node create <satellite_hostname>

The storage pool can also be created as usual in LINSTOR. Only the name of the previously registered Exos enclosure as well as the serial number of the Exos pool needs to be specified:

# linstor storage-pool create exos \
  <satellite_hostname> <linstor_pool_name> <exos_encl_name> <exos_pool_sn>

the linstor_pool_name can be set to (almost) any unique string for the LINSTOR deployment.

Here is an example of mapping an Exos Pool in Exos enclosure Alpha to two Satellite nodes:

# linstor storage-pool create exos \
   node1 poolA Alpha 00c0ff29a5f5000095a2075d01000000
# linstor storage-pool create exos \
   node2 poolA Alpha 00c0ff29a5f5000095a2075d01000000

After creating an exos storage pool the LINSTOR Satellite will scan the given Exos enclosure for connected ports. If cabled, these ports will be listed in the following command:

# linstor exos map -p +----------------------------------------------+ |
Node Name | Enclosure Name | Connected Ports |
|==============================================|
| node1     | Alpha          | A0, B0          |
| node2     | Alpha          | A1, B1          |
+----------------------------------------------+

The pool configuration is shown by:

hr01u09:~ # linstor sp list -s poolA -p
+----------------------------------------------------------------------------------------------+
| StoragePool | Node  | Driver   | PoolName                               | FreeCapacity | ... |
|==============================================================================================|
| poolA       | node1 | EXOS     | Alpha_00c0ff29a5f5000095a2075d01000000 |      581 TiB | ... |
| poolA       | node2 | EXOS     | Alpha_00c0ff29a5f5000095a2075d01000000 |      581 TiB | ... |
+----------------------------------------------------------------------------------------------+

Detailed description of all the available Exos commands is found with built-in help.

# linstor exos -h

11.3. Creating Resources Backed by Exos Storage Pools

Creating LINSTOR resources from Exos backed storage-pools follows normal LINSTOR usage patterns as described in other sections of the LINSTOR User’s Guide such as the sections describing LINSTOR resource groups or the more granular resource-definition, volume-definition, resource creation workflow.

12. LINSTOR Volumes in CloudStack

This chapter describes using LINSTOR to provision volumes that can be used to back primary storage in Apache CloudStack. CloudStack primary storage stores the virtual disks for virtual machines (VMs) running on hosts in CloudStack. A LINBIT-developed CloudStack plug-in integrates LINSTOR with CloudStack. A benefit to integrating LINSTOR with CloudStack is that it can be a way to provide highly available primary storage that is also flexible to manage.

Currently, the LINSTOR plug-in for CloudStack can only be used to provision volumes for use with KVM hypervisors.

Setting up and deploying CloudStack can be a complex task. A production-ready deployment can take several weeks to months before it is ready for users. A basic test deployment in a virtual environment can be set up in a few hours perhaps. This chapter will deal only with aspects related to integrating LINSTOR in CloudStack and should be considered a general overview. You should supplement instructions in this chapter with instructions and best practice recommendations from the CloudStack documentation.

Attention should be paid to security, firewall, and resource provisioning instructions in the CloudStack documentation, and in other chapters in the LINSTOR User’s Guide, before production deployment.
In this chapter, as in other areas of the LINSTOR User’s Guide, the word node is used. In most cases, you can think of a node as equivalent to a CloudStack host.

12.1. Introduction to CloudStack

From the CloudStack documentation: “Apache CloudStack is an open source Infrastructure-as-a-Service (IaaS) platform that manages and orchestrates pools of storage, network, and computer resources to build a public or private IaaS compute cloud.”

12.2. Preparing Your Environment For CloudStack and LINSTOR Deployment

You will need to make a few preparatory steps before deploying LINSTOR for use with CloudStack.

12.2.1. Configuring Time Synchronization

It is important that the nodes in your cluster are time synchronized. To do this, you can install and configure a tool such as Chrony or OpenNTPD.

12.2.2. Adding Node IP Addresses and Host Names

Add your cluster nodes’ IP addresses and host names to each node’s /etc/hosts file.

12.2.3. Setting a Root User Password and Permitting SSH Login

Set a root password by using the passwd root command. Then edit your SSH configuration file (/etc/ssh/sshd_config) to permit root login. The line in your file should look like this:

PermitRootLogin yes

12.2.4. Configuring SELinux

For CloudStack to work properly, SELinux must be set to permissive or disabled (not recommended in production).

12.3. Installing and Preparing LINSTOR for CloudStack

The LINSTOR plug-in is included in Apache CloudStack versions 4.16.1 and later. You do not have to install anything else to support LINSTOR. CloudStack versions 4.16.0 and earlier do not support the LINSTOR plug-in.

One of the pull requests did not merge properly in CloudStack v4.17.0 which caused a CloudStack UI bug in the CloudStack initialization wizard. More details are available here. If you need to be on the v4.17 branch, it is recommended (at time of writing) that you install v4.17.1.

Follow the installation instructions in the LINSTOR User’s Guide to install LINSTOR on the storage providing nodes in your cluster.

A basic outline of installation steps is:

  1. Install necessary packages for storage layers that you will be using, for example, ZFS or LVM. The steps below use ZFS as a backing storage layer for LINSTOR.

  2. Install the necessary LINSTOR packages (DRBD kernel module, linbit-sds-controller, and linbit-sds-satellite packages) from LINBIT repositories if you are a LINBIT customer, otherwise, you will need to build from source.

  3. Restart the multipathd daemon.

    systemctl restart multipathd
  4. Enable and start the LINSTOR Controller and LINSTOR Satellite services on your nodes.

    # systemctl enable --now linstor-controller
    # systemctl enable --now linstor-satellite
  5. Add your nodes to LINSTOR.

    linstor node create <node_host_name>
  6. Create a new LINSTOR storage pool on all of your participating nodes. For example, given a ZFS pool named zfs_storage, enter the following to create a storage pool named DfltStorPool:

    # linstor storage-pool create zfs <node_host_name> DfltStorPool zfs_storage
  7. Create a LINSTOR resource group to be used for CloudStack. To create a resource group named cloudstack, to be placed on two of your cluster nodes, enter:

    # linstor resource-group create cloudstack --place-count 2 --storage-pool DfltStorPool
  8. Create a LINSTOR volume group from your resource group, by entering the command:

    # linstor volume-group create cloudstack

12.3.1. Verifying Creating Resources

After installing LINSTOR and creating a resource group backed by a storage pool and storage layer, test that you can create storage resources. You can do this by spawning resources from the resource group that you created.

The CloudStack setup best practices recommend that a primary storage mount point (and therefore the LINSTOR resource that backs it) “should not exceed 6TB in size.”
# linstor resource-group spawn cloudstack testres 1GiB

Verify that LINSTOR created your resources by using a resource list command.

# linstor resource list
+----------------------------------------------------------------------------------+
| ResourceName | Node   | Port | Usage  | Conns |      State | CreatedOn           |
|-=================================================================================|
| testres      | node-0 | 7000 | Unused | Ok    |   UpToDate | 2022-11-10 20:12:30 |
| testres      | node-1 | 7000 | Unused | Ok    |   UpToDate | 2022-11-10 20:12:30 |
| testres      | node-2 | 7000 | Unused | Ok    | TieBreaker | 2022-11-10 20:12:29 |
+----------------------------------------------------------------------------------+

12.3.2. Cleaning Up After Verifying Resource Creation

After verifying resource creation, you can delete the resource, here testres, in your cluster by entering:

# linstor resource-definition delete testres

12.4. Installing CloudStack

After installing and preparing LINSTOR, you can install and configure CloudStack. As disclaimed previously, you should take these instructions as a way to setup CloudStack quickly for testing and illustrative purposes. Refer to CloudStack documentation for detailed instructions and best practice recommendations, before deploying into production.

12.4.1. Installing MySQL

First, install a MySQL server instance that is necessary for CloudStack’s database.

On Ubuntu, enter:

# apt install -y mysql-server

On RHEL, enter:

# dnf install -y mysql-server

12.4.2. Configuring the CloudStack Database

After installing the MySQL server package, create a CloudStack database configuration file named /etc/mysql/conf.d/cloudstack.cnf with the following contents:

[mysqld]
innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=350 (1)
log-bin=mysql-bin
binlog-format = 'ROW'
1 350 is the max_connections value specified in the CloudStack installation guide. You can change this value depending on your needs.

If you are on an Ubuntu 16.04 or later system, for binary logging, you need to specify a server_id in your .cnf database configuration file, for example:

[mysqld]
server_id = 1
innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=350
log-bin=mysql-bin
binlog-format = 'ROW'

Then restart the MySQL service by entering systemctl restart mysql.

12.4.3. Installing NFS for Secondary Storage

Next, install and configure NFS for CloudStack’s secondary storage. You only need to do this on the node that will be your CloudStack management node. CloudStack uses secondary storage to store such things as operating system images for VMs and snapshots of VM data.

To install NFS, on Ubuntu, enter:

# apt install -y nfs-kernel-server

On RHEL, enter:

# dnf install -y nfs-utils

After installing the NFS server, create an NFS export for CloudStack’s secondary storage by entering the following commands:

# mkdir -p /export/secondary
# echo "/export *(rw,async,no_root_squash,no_subtree_check)" >> /etc/exports
# exportfs -a

Next, enable and start the NFS server service.

# systemctl enable --now nfs-server

12.5. Installing and Configuring CloudStack

General CloudStack installation and configuration instructions follow. As your environment may have specific needs or variations, you should also reference the instructions in the CloudStack Installation Guide.

12.5.1. Installing CloudStack

While official CloudStack releases are “always in source code form,” for convenience, there are community generated DEB and RPM packages available at cloudstack.org:

You can follow the links above to find and download the packages that you need for your installation. Be sure to verify the integrity of downloaded packages against CloudStack’s signing keys, as outlined in the instructions here.

Alternatively, you can follow instructions here to configure the CloudStack repository appropriate to your Linux distribution and then pull and install packages by using your distribution’s package manager.

After adding the CloudStack repository, you may need to update the package manager’s repository list, before you can install packages.

For your CloudStack management node, install these packages:

  • cloudstack-management

  • cloudstack-common

  • cloudstack-ui

For your other cluster nodes that will be hosting VMs, install the cloudstack-agent package.

12.5.2. Initializing the CloudStack Database

After installing the necessary CloudStack packages, initialize the CloudStack database.

For testing purposes, you can enter the following command on your management node:

# cloudstack-setup-databases cloud:cloud --deploy-as=root:nonsense -i <node_name>

Here, the cloud after cloud: and nonsense are passwords that you can change as you see fit.

For production deployments, follow the more detailed instructions in the CloudStack Installation Guide.

12.6. Installing the CloudStack System Virtual Machine Image Template

CloudStack needs to run some system VMs for some of its functionality. You can download a CloudStack VM template image and then run a CloudStack script that will prepare the image for various system VMs in deployment. On the CloudStack management node, enter the following commands:

# CS_VERSION=4.17
# CS_VERSION_PATCH=4.17.1
# wget https://download.cloudstack.org/systemvm/$CS_VERSION/systemvmtemplate-$CS_VERSION_PATCH-kvm.qcow2.bz2
# /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt \
-m /export/secondary \
-f systemvmtemplate-$CS_VERSION_PATCH=-kvm.qcow2.bz2 \
-h kvm -o localhost -r cloud -d cloud

12.7. Configuring KVM Hypervisor Hosts for Use in CloudStack

Currently, the LINSTOR CloudStack plug-in only supports KVM hypervisor hosts. The instructions that follow are for configuring your CloudStack installation with KVM hypervisor hosts.

Enter the following command to add libvirt configurations to every node in your cluster that will host CloudStack VMs:

# cat << EOF >> /etc/libvirt/libvirtd.conf
listen_tls = 0
listen_tcp = 1
tcp_port = "16509"
auth_tcp = "none" # not suitable for production
mdns_adv = 0
EOF

Restart the libvirtd service on all hypervisor nodes.

# systemctl restart libvirtd

12.7.1. Configuring AppArmor

If you are running CloudStack on Ubuntu Linux and if AppArmor is enabled, enter the following:

# ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/
# ln -s /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper /etc/apparmor.d/disable/
# apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd
# apparmor_parser -R /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper

12.7.2. Restarting the CloudStack Management Service

After making the necessary setup and preparatory configurations, restart the cloudstack-management service.

# systemctl restart cloudstack-management

You can follow the progress of CloudStack’s initial database setup by entering:

# journalctl -u cloudstack-management -f

12.7.3. Logging into the CloudStack UI

After some time, you should be able to log into the CloudStack management UI. Given a management node resolvable host name of node-0, enter the following URL into a web browser on a computer in your cluster’s network: http://node-0:8080/client.

Once you are greeted by the CloudStack UI’s portal login page, log into the portal by using the default user name admin and the default password password.

After successfully logging in, the CloudStack UI will display the “Hello and Welcome to CloudStack” page.

12.7.4. Running the CloudStack Initialization Wizard

You can continue to set up CloudStack by launching an initialization wizard. Click on the “Continue with installation” button to launch the wizard.

The wizard will first prompt you to change the default password for the administrator user. After changing the password, you can continue through the wizard steps to configure a zone, network, and resources details. Complete the fields in each setup step according to your environment and needs. More details about initializing CloudStack can be found here.

The following fields will be common to all LINSTOR use cases in CloudStack:

  • Zone details:

    • Hypervisor: KVM

  • Add resources, IP Address step:

    • Host Name: <host_name_of_cluster_node_that_will_host_VMs>

    • Username: root

    • Password: <root_password_that_you_configured_previously_for_the_host>

  • Add resources, Primary Storage step:

    • Protocol: Linstor

    • Server: <IP_address_of_LINSTOR_controller_node>

    • Resource Group: <LINSTOR_resource_group_name_that_you_configured_previously>

Based on configuring an NFS export for secondary storage earlier, complete the fields presented during the “Add resources, Secondary Storage” step as follows:

  • Provider: NFS

  • IP Address: <IP_address_of_NFS_server> # should be the CloudStack management node

  • Path: <NFS_mount_point> # /export/secondary, as configured previously

After completing entry fields in the “Add resources” fields and clicking the “Next” button, the wizard will display a message indicating the “Zone is ready to launch.” Click on the “Launch Zone” button.

The “Adding Host” step of the “Launch Zone” process may take a while.

After the zone is added, the wizard will show a “Zone creation complete” message. You can then click on the “Enable Zone” button. After another “Success” notification you will be returned to the CloudStack UI dashboard.

12.7.5. Verifying Primary Storage in CloudStack

Click on the “Infrastructure” icon on the left of the screen, then click on “Primary Storage”. The “Primary Storage” screen should show your LINSTOR-backed primary storage pool in an “Up” state.

12.7.6. Verifying Secondary Storage in CloudStack

Click on the “Infrastructure” icon on the left of the screen, then click on the “Secondary Storage” link. The “Secondary Storage” screen should show your NFS-backed secondary storage in a “ReadWrite” access state.

12.8. Taking Next Steps in CloudStack

After configuring LINSTOR for use in CloudStack you can move onto other tasks, such as adding hosts to host your CloudStack VMs.

LINBIT has also made available a video demonstrating deploying LINSTOR and CloudStack into a three-node VM cluster. You can view the video here.

13. LINSTOR GUI

LINSTOR GUI is a LINSTOR client alternative, currently in Technology Preview phase. This component is proprietary and users need to have access to LINBIT customer-only repositories to be able to use it.

13.1. Prerequisites

  • Access to LINBIT’s customer repositories.

  • Running and working LINSTOR controller instance.

13.2. Installing the LINSTOR GUI

Install LINSTOR GUI package on the same node as the LINSTOR controller and restart the linstor-controller service.

On yum/dnf based distributions you can install the software by entering the command:

yum install linstor-gui

On apt based distributions you install the software by entering the command:

apt install linstor-gui

On Kubernetes, LINSTOR GUI is a built-in feature since linstor-controller v1.15.0.

13.3. Administering the LINSTOR Cluster

Open your web browser and navigate to http://CONTROLLER_IP:3370/ui/ (the last / is mandatory) to manage your LINSTOR cluster.


1. If a host is also a storage node, it will use a local copy of an image if that is available