LINSTOR was originally developed to manage DRBD resources. While you can still use LINSTOR to make managing DRBD more convenient, LINSTOR has evolved and it is often integrated with software stacks higher up to provide persistent storage more easily and more flexibly than would otherwise be possible within those stacks.

LINSTOR can be used on its own or you can integrate it with other platforms, such as Kubernetes, OpenShift, OpenNebula, OpenStack, Proxmox VE, and others. LINSTOR runs on bare-metal on-premise hardware, or you can use it within virtual machines (VMs), containers, clouds, or hybrid environments.

LINSTOR can work with the following storage provider and related technologies:

By using LINSTOR, you can work with these technologies on their own or else in various meaningful combinations.

The linstor-controller service relies on a database that holds all the configuration information for the whole cluster. A node or container running the LINSTOR controller software is responsible for resource placement, resource configuration, and orchestration of any operational processes that require a view of the whole cluster.

Multiple controllers can be used for LINSTOR, for example, when setting up a highly available LINSTOR cluster, but only one controller can be active.

As mentioned earlier, the LINSTOR controller operates on a separate plane from the data plane that it manages. You can stop the controller service, update or reboot the controller node, and still have access to your data hosted on the LINSTOR satellite nodes. While you can still access and serve the data on your LINSTOR satellite nodes, without a running controller node, you will not be able to perform any LINSTOR status or management tasks on the satellite nodes.

The linstor-satellite service runs on each node where LINSTOR consumes local storage or provides storage to services. It is stateless and receives all the information it needs from the node or container running the LINSTOR controller service. The LINSTOR satellite service runs programs like lvcreate and drbdadm. It acts like an agent on a node or in a container that carries out instructions that it receives from the LINSTOR controller node or container.

When you need to interface with LINSTOR, you can send instructions to the active LINSTOR controller by using one of its user interfaces (UIs): the LINSTOR client, or the LINBIT SDS GUI.

Both of these UIs rely on the LINSTOR REST API.

LINSTOR takes an object-oriented approach to software-defined storage (SDS). LINSTOR objects are the end result that LINSTOR presents to the user or application to consume or build upon.

The most commonly used LINSTOR objects are explained below and a full list of objects follows.

While definitions and groups are also LINSTOR objects, they are a special kind. Definition and group objects can be thought of as profiles or templates. These template objects are used to create child objects that will inherit their parent object’s attributes. They might also have attributes that can affect child objects but are not attributes of the child objects themselves. These attributes could be things such as the TCP port to use for DRBD replication or the volume number that a DRBD resource should use.

The following are some general rules for object hierarchy in LINSTOR:

This section uses diagrams to represent the hierarchical relationships between some of the most frequently used LINSTOR objects. Because of the number of LINSTOR objects and their interconnectedness, multiple diagrams are shown first rather than a single diagram, to simplify the conceptualization.

The next diagram shows the relationships between LINSTOR group objects on a single satellite node.

While the two preceding diagrams show higher-lower relationships between common LINSTOR objects, you can also think of some LINSTOR objects as having parent-child relationships. The next diagram introduces this kind of relationship between LINSTOR objects by using a storage pool definition (parent object) and a storage pool (child object) as an example. A parent object can have multiple child objects, as shown in the following diagram.

Having introduced the concept of parent-child relationships in a conceptual diagram, the next diagram is a modified version of the second diagram with some of those relationships added for groups and definitions. This modified diagram also incorporates some of the higher-lower relationships that were shown in the first diagram.

The next diagram synthesizes the relationship concepts of the preceding diagrams while also introducing new LINSTOR objects related to snapshots and connections. With the many objects and criss-crossing lines, the reason for building up to this diagram should be apparent.

Even with its seeming complexity, the preceding diagram is still a simplification and not an all-encompassing representation of the possible relationships between LINSTOR objects. As listed earlier, there are more LINSTOR objects than are shown in the diagram^[3].

The good news is that you do not need to memorize the preceding diagram to work with LINSTOR. It could be useful to refer to though if you are trying to troubleshoot attributes that you have set on LINSTOR objects and their inheritance and effects on other LINSTOR objects in your cluster.

LINSTOR is packaged in both the RPM and the DEB variants:

For further details about these packages see the Installable Components section above.

You can configure LINSTOR to encrypt storage volumes, by using LUKS (dm-crypt), as detailed in the Encrypted Volumes section of this user guide. Refer to the LUKS and dm-crypt projects for FIPS compliance status.

You can also configure LINSTOR to encrypt communication traffic between a LINSTOR satellite and a LINSTOR controller, by using SSL/TLS, as detailed in the Secure Satellite Connections section of this user guide.

LINSTOR can also interface with Self-Encrypting Drives (SEDs) and you can use LINSTOR to initialize an SED drive. LINSTOR stores the drive’s password as a property that applies to the storage pool associated with the drive. LINSTOR encrypts the SED drive password by using the LINSTOR master passphrase that you must create first.

By default, LINSTOR uses the following cryptographic algorithms:

A FIPS compliant version of LINSTOR is available for the use cases mentioned in this section. If you or your organization require FIPS compliance at this level, contact [email protected] for details.

To use LINSTOR to create storage volumes, you will need to install a volume manager, either LVM or ZFS, if one is not already installed on your system.

If you are a LINBIT® customer, you can download a LINBIT created helper script and run it on your nodes to:

Enabling LINBIT package repositories will give you access to LINBIT software packages, DRBD® kernel modules, and other related software such as cluster managers and OCF scripts. You can then use a package manager to fetch, install, and manage updating installed packages.

After registering your node and enabling the drbd-9 LINBIT package repository, you can use a DEB, RPM, or YaST2 based package manager to install LINSTOR and related components.

The LINSTOR project’s GitHub page is here: https://github.com/LINBIT/linstor-server.

LINBIT also has downloadable archived files of source code for LINSTOR, DRBD, and more, available here: https://linbit.com/linbit-software-download-page-for-linstor-and-drbd-linux-driver/.

Before upgrading LINSTOR, you should back up the LINSTOR controller database. If you are using the embedded default H2 database and you upgrade the linstor-controller package, an automatic backup file of the database will be created in the default /var/lib/linstor directory. This file is a good restore point if a linstor-controller database migration should fail. It is recommended to report the error to LINBIT and restore the old database file and roll back to your previous controller version.

If you use any external database or etcd, make a manual backup of your current database, before upgrading LINSTOR. You can use this backup as a restore point if needed.

On RPM-based Linux distributions, LINSTOR services are not automatically restarted when you upgrade LINSTOR packages. For this reason, the upgrading process does not entail as much downtime for your LINSTOR services (LINSTOR control plane) as it might on other systems.

On all nodes, update applicable LINSTOR packages. Modify the command below to only update packages applicable to each node’s role in your LINSTOR cluster:

After updating LINSTOR packages on all nodes, as simultaneously as you can, restart the LINSTOR satellite service on all your satellite nodes, except for the actively running controller node if that node is in a combined role.

Next, restart the LINSTOR controller service on the node where the service is actively running. If that node is in a combined role, also update the LINSTOR satellite service, for example:

On DEB-based Linux systems, because LINSTOR services will restart automatically after installing an updated version, you will need to plan for more downtime of the LINSTOR control plane than on an RPM-based system.

First, upgrade the LINSTOR controller software on the active LINSTOR controller node.

Upgrading the software will automatically restart the controller service. At this point, if you enter a linstor node list command, output will show that your satellite nodes are in an OFFLINE(VERSION MISMATCH) state.

Next, update the LINSTOR client and satellite packages on nodes where those packages are installed. Modify the command below to only update packages applicable to each node’s role in your LINSTOR cluster:

Uprading the LINSTOR satellite software package on your satellite (and combined) nodes will automatically restart the LINSTOR satellite service on those nodes.

After upgrading all LINSTOR packages applicable to each node in your cluster, verify:

Alternatively, you can create the /etc/linstor/linstor-client.conf file and add a controllers= line in the global section.

If you have multiple LINSTOR controllers configured you can simply specify them all in a comma-separated list. The LINSTOR client will try them in the order listed.

You can use LINSTOR client commands in a much faster and convenient way by only entering the starting letters of the commands, subcommands, or parameters. For example, rather than entering linstor node list you can enter the LINSTOR short notation command linstor n l.

Entering the command linstor commands will show a list of possible LINSTOR client commands along with the abbreviated notation for each command. You can use the --help flag with any of these LINSTOR client commands to get the abbreviated notation for the command’s subcommands.

If you specify an IP address when you create a LINSTOR node, you can give your node an arbitrary name. The LINSTOR client will show an INFO message about this when you create the node:

LINSTOR will automatically detect the created node’s local uname --nodename which will be later used for DRBD resource configurations, rather than the arbitrary node name. To avoid confusing yourself and possibly others, in most cases, it would make sense to just use a node’s hostname when creating a LINSTOR node.

When you use linstor node list LINSTOR will show that the new node is marked as offline. Now start and enable the LINSTOR satellite service on the new node so that the service comes up on reboot as well:

About 10 seconds later you will see the status in linstor node list as online. Of course the satellite process might be started before the controller knows about the existence of the satellite node.

If you want to have other services wait until the LINSTOR satellite service had a chance to create the necessary devices (that is, after a boot), you can update the corresponding .service file and change Type=simple to Type=notify.

This will cause the satellite to delay sending the READY=1 message to systemd until the controller connects, sends all required data to the satellite and the satellite at least tried once to get the devices up and running.

When you create a LINSTOR node, you can also specify a node type. Node type is a label that indicates the role that the node serves within your LINSTOR cluster. Node type can be one of controller, auxiliary, combined, or satellite. For example to create a LINSTOR node and label it as a controller and a satellite node, enter the following command:

The --node-type argument is optional. If you do not specify a node type when you create a node, LINSTOR will use a default type of satellite.

If you want to change a LINSTOR node’s assigned type after creating the node, you can enter a linstor node modify --node-type command.

Before you use LINSTOR to create storage objects in your cluster, or in case you are in a troubleshooting situation, you can list the storage providers and storage layers that are supported on the satellite nodes in your cluster. To do this, you can use the node info command.

Output from the command will show two tables. The first table will show the available LINSTOR storage provider back ends. The second table will show the available LINSTOR storage layers. Both tables will indicate whether a given node supports the storage provider or storage layer. A plus sign, +, indicate “is supported”. A minus sign, -, indicates “is not supported”.

Example output for a cluster might be similar to the following:

On each host contributing storage, you need to create either an LVM volume group (VG) or a ZFS zPool. The VGs and zPools identified with one LINSTOR storage pool name might have different VG or zPool names on the hosts, but do yourself a favor, for coherency, use the same VG or zPool name on all nodes.

After creating a volume group on each of your nodes, you can create a storage pool that is backed by the volume group on each of your nodes, by entering the following commands:

To list your storage pools you can enter:

or using LINSTOR abbreviated notation:

In clusters where you have only one kind of storage and the capability to hot swap storage devices, you might choose a model where you create one storage pool per physical backing device. The advantage of this model is to confine failure domains to a single storage device.

Both the Exos and LVM2 storage providers offer the option of multiple server nodes directly connected to the storage array and drives. You can use the LINSTOR storage-pool create command option --shared-space when configuring a LINSTOR pool to use the same LVM2 volume group accessible by two or more nodes.

The following example shows using the LVM2 volume group UUID as the shared space identifier for a storage pool accessible by nodes alpha and bravo:

This example also uses the --external-locking option. By using this option, you tell LINSTOR not to use its internal locking mechanism for shared storage pools but rather use LVM’s mechanism in this example.

You can omit both the --shared option for vgcreate and the --external-locking option for linstor storage-pool create. If you do this, LINSTOR will use its own mechanism to take care that only one node in the same --shared-space grouping will use LVM. Using LINSTOR’s internal mechanism to lock LVM use will not however limit the access to single volumes. Neither LINSTOR’s nor LVM’s locking mechanism, for example, will limit DRBD I/O.

Exos pools will use the Exos pool serial number by default for the --shared-space identifier.

Since linstor-server 1.5.2 and a recent linstor-client, LINSTOR can create LVM/ZFS pools on a satellite for you. The LINSTOR client has the following commands to list possible disks and create storage pools, but such LVM/ZFS pools are not managed by LINSTOR and there is no delete command, so such action must be done manually on the nodes.

Will give you a list of available disks grouped by size and rotational(SSD/Magnetic Disk).

It will only show disks that pass the following filters:

With the create-device-pool command you can create a LVM pool on a disk and also directly add it as a storage pool in LINSTOR.

If the --storage-pool option was provided, LINSTOR will create a storage pool with the given name.

For more options and exact command usage refer to the LINSTOR client --help text.

With some setup and configuration, you can use storage pools of different storage provider types to back a LINSTOR resource. This is called storage pool mixing. For example, you might have a storage pool on one node that uses an LVM thick-provisioned volume while on another node you have a storage pool that uses a thin-provisioned ZFS zpool.

Because most LINSTOR deployments will use homogenous storage pools to back resources, storage pool mixing is only mentioned here so that you know that the feature exists. It might be a useful feature when migrating storage resources, for example. You can find further details about this, including prerequisites, in Mixing Storage Pools of Different Storage Providers.

LINSTOR has the following supported storage plugins at the time of writing:

With the resource create command you can assign a resource definition to named nodes explicitly.

When you create (spawn) a resource from a resource group, it is possible to have LINSTOR automatically select nodes and storage pools to deploy the resource to. You can use the arguments mentioned in this section to specify constraints when you create or modify a resource group. These constraints will affect how LINSTOR automatically places resources that are deployed from the resource group.

You can delete a resource definition by using the command:

This will remove the named resource definition from the entire LINSTOR cluster. The resource is removed from all nodes and the resource entry is marked for removal from LINSTOR’s database tables. After LINSTOR has removed the resource from all the nodes, the resource entry is removed from LINSTOR’s database tables.

You can delete a resource using the command:

Unlike deleting a resource definition, this command will only delete a LINSTOR resource from the node (or nodes) that you specify. The resource is removed from the node and the resource entry is marked for removal from LINSTOR’s database tables. After LINSTOR has removed the resource from the node, the resource entry is removed from LINSTOR’s database tables.

Deleting a LINSTOR resource might have implications for a cluster, beyond just removing the resource. For example, if the resource is backed by a DRBD layer, removing a resource from one node in a three node cluster could also remove certain quorum related DRBD options, if any existed for the resource. After removing such a resource from a node in a three node cluster, the resource would no longer have quorum as it would now only be deployed on two nodes in the three node cluster.

After running a linstor resource delete command to remove a resource from a single node, you might see informational messages such as:

Also unlike deleting a resource definition, you can delete a resource while there are existing snapshots of the resource’s storage pool. Any existing snapshots for the resource’s storage pool will persist.

You can delete a resource group by using the command:

As you might expect, this command deletes the named resource group. You can only delete a resource group if it has no associated resource definitions, otherwise LINSTOR will present an error message, such as:

To resolve this error so that you can delete the resource group, you can either delete the associated resource definitions, or your can move the resource definitions to another (existing) resource group:

You can find which resource definitions are associated with your resource group by entering the following command:

To backup the LINSTOR database to a new file named db_export in your home directory, enter the following commands:

After backing up the database, you can copy the backup file to a safe place.

The resulting database backup is a plain JSON document, containing not just the actual data, but also some metadata about when the backup was created, from which database, and other information.

Restoring the database from a previously made backup is similar to export-db from the previous section.

For example, to restore the previously made backup from the db_export file, enter the following commands:

You can only import a database from a previous backup if the currently installed version of LINSTOR is the same (or higher) version as the version that you created the backup from. If the currently installed LINSTOR version is higher than the version that the database backup was created from, when you import the backup the data will be restored with the same database scheme of the version used during the export. Then, the next time that the controller starts, the controller will detect that the database has an old scheme and it will automatically migrate the data to the scheme of the current version.

Since the exported database file contains some metadata, an exported database file can be imported into a different database type than it was exported from.

This allows the user to convert, for example, from an etcd setup to an SQL based setup. There is no special command for converting the database format. You only have to specify the correct linstor.toml configuration file by using the --config-directory argument (or updating the default /etc/linstor/linstor.toml and specifying the database type that you want to use before importing). See the LINSTOR User Guide for more information about specifying a database type. Regardless of the type of database that the backup was created from, it will be imported in the database type that is specified in the linstor.toml configuration file.

To configure the highly available storage, you can use LINSTOR itself. One of the benefits of having the storage under LINSTOR control is that you can easily extend the HA storage to new cluster nodes.

After creating the linstor_db resource, you can move the LINSTOR database to the new storage and create a systemd mount service. First, stop the current controller service and disable it, as it will be managed by DRBD Reactor later.

Next, create the systemd mount service.

Copy the /etc/systemd/system/var-lib-linstor.mount mount file to all the cluster nodes that you want to have the potential to run the LINSTOR controller service (standby controller nodes). Again, do not systemctl enable any of these services because DRBD Reactor will manage them.

The next step is to install LINSTOR controllers on all nodes that have access to the linstor_db DRBD resource (as they need to mount the DRBD volume) and which you want to become a possible LINSTOR controller. It is important that the controllers are manged by drbd-reactor, so verify that the linstor-controller.service is disabled on all nodes! To be sure, execute systemctl disable linstor-controller on all cluster nodes and systemctl stop linstor-controller on all nodes except the one it is currently running from the previous step. Also verify that you have set chattr +i /var/lib/linstor on all potential controller nodes if you use LINSTOR version equal or greater to 1.14.0.

For starting and stopping the mount service and the linstor-controller service, use DRBD Reactor. Install this component on all nodes that could become a LINSTOR controller and edit their /etc/drbd-reactor.d/linstor_db.toml configuration file. It should contain an enabled promoter plugin section like this:

Depending on your requirements you might also want to set an on-stop-failure action and set stop-services-on-exit.

After that restart drbd-reactor and enable it on all the nodes you configured it.

Check that there are no warnings from drbd-reactor service in the logs by running systemctl status drbd-reactor. As there is already an active LINSTOR controller things will just stay the way they are. Run drbd-reactorctl status linstor_db to check the health of the linstor_db target unit.

The last but nevertheless important step is to configure the LINSTOR satellite services to not delete (and then regenerate) the resource file for the LINSTOR controller DB at its startup. Do not edit the service files directly, but use systemctl edit. Edit the service file on all nodes that could become a LINSTOR controller and that are also LINSTOR satellites.

After this change you should execute systemctl restart linstor-satellite on all satellite nodes.

NVMe-oF/NVMe-TCP allows LINSTOR to connect diskless resources to a node with the same resource where the data is stored over NVMe fabrics. This leads to the advantage that resources can be mounted without using local storage by accessing the data over the network. LINSTOR is not using DRBD in this case, and therefore NVMe resources provisioned by LINSTOR are not replicated, the data is stored on one node.

To use NVMe-oF/NVMe-TCP with LINSTOR the package nvme-cli needs to be installed on every node which acts as a satellite and will use NVMe-oF/NVMe-TCP for a resource. For example, on a DEB-based system, to install the package, enter the following command:

要使资源使用NVMe-oF/NVMe-TCP，必须在创建资源定义时提供一个附加参数：

To use NVMe-TCP rather than the default NVMe-oF, the following property needs to be set:

The property NVMe/TRType can alternatively be set on resource-group or controller level.

Next, create the volume-definition for your resource:

在节点上创建资源之前，必须知道数据将在本地存储在哪里，以及哪个节点通过网络访问它。

First, create the resource on the node where your data will be stored:

在将通过网络访问资源数据的节点上，必须将资源定义为无盘：

现在，您可以在一个节点上装载资源 nvmedata。

A DM-Writecache device is composed of two devices: one storage device and one cache device. LINSTOR can setup such a writecache device, but needs some additional information, like the storage pool and the size of the cache device.

The two properties set in the examples are mandatory, but can also be set on controller level which would act as a default for all resources with WRITECACHE in their --layer-list. However, note that the Writecache/PoolName refers to the corresponding node. If the node does not have a storage pool named pmempool you will get an error message.

The four mandatory parameters required by DM-Writecache are either configured through a property or figured out by LINSTOR. The optional properties listed in the mentioned link can also be set through a property. Refer to linstor controller set-property --help for a list of Writecache/* property keys.

使用 --layer-list DRBD,WRITECACHE,STORAGE 将DRBD配置为使用外部元数据时，只有备份设备将使用writecache，而不是保存外部元数据的设备。

LINSTOR can also setup a DM-Cache device, which is very similar to the DM-Writecache from the previous section. The major difference is that a cache device is composed by three devices: one storage device, one cache device and one meta device. The LINSTOR properties are quite similar to those of the writecache but are located in the Cache namespace:

Refer to linstor controller set-property --help for a list of Cache/* property keys and default values for omitted properties.

Using --layer-list DRBD,CACHE,STORAGE while having DRBD configured to use external metadata, only the backing device will use a cache, not the device holding the external metadata.

The storage layer will provide new devices from well known volume managers like LVM, ZFS or others. Every layer combination needs to be based on a storage layer, even if the resource should be diskless – for that type there is a dedicated diskless provider type.

For a list of providers with their properties refer to Storage Providers.

For some storage providers LINSTOR has special properties:

While LINSTOR resources are most often backed by storage pools that consist of only one storage provider type, it is possible to user storage pools of different types to back a LINSTOR resource. This is called mixing storage pools. This might be useful when migrating storage resources but could also have some consequences. Read this section carefully before configuring mixed storage pools.

A storage pool with the type remote SPDK can only be created on a “special satellite”. For this you first need to start a new satellite using the command:

This will start a new satellite instance running on the same machine as the controller. This special satellite will do all the REST based RPC communication towards the remote SPDK proxy. As the help message of the LINSTOR command shows, the administrator might want to use additional settings when creating this special satellite:

The difference between the --api-* and their corresponding --api-\*-env versions is that the version with the -env ending will look for an environment variable containing the actual value to use whereas the --api-\* version directly take the value which is stored in the LINSTOR property. Administrators might not want to save the --api-pw in plain text, which would be clearly visible using commands like linstor node list-property <nodeName>.

Once that special satellite is up and running the actual storage pool can be created:

Whereas node_name is self-explanatory, name is the name of the LINSTOR storage pool and driver_pool_name refers to the SPDK logical volume store.

Once this remotespdk storage pool is created the remaining procedure is quite similar as using NVMe: First the target has to be created by creating a simple “diskful” resource followed by a second resource having the --nvme-initiator option enabled.

To use multiple network paths for DRBD setups, the PrefNic property is not sufficient. Instead the linstor node interface and linstor resource-connection path commands should be used, as shown below.

The first four commands in the example define a network interface (nic1 and nic2) for each node (alpha and bravo) by specifying the network interface’s IP address. The last two commands create network path entries in the DRBD .res file that LINSTOR generates. This is the relevant part of the resulting .res file:

下面是有关命令的详细信息。

Before LINSTOR can encrypt any volume a master passphrase needs to be created. This can be done with the LINSTOR client.

`crypt create-passphrase`将等待用户输入初始主密码（因为所有其他crypt命令都没有参数）。

如果您想更改主密码，可以使用以下方法：

luks 层可以在创建资源定义或资源本身时添加，而建议使用前一种方法，因为它将自动应用于从该资源定义创建的所有资源。

要输入主密码（在controller重新启动后），请使用以下命令：

It is possible to automate the process of creating and re-entering the master passphrase.

To use this, either an environment variable called MASTER_PASSPHRASE or an entry in /etc/linstor/linstor.toml containing the master passphrase has to be created.

The required linstor.toml looks like this:

If either one of these is set, then every time the controller starts it will check whether a master passphrase already exists. If there is none, it will create a new master passphrase as specified. Otherwise, the controller enters the passphrase.

Some evacuation cases might need special planning. For example, if you are evacuating more than one node, you can exclude the nodes from participating in LINSTOR’s resource autoplacer. You can do this by using the following command on each node that you want to evacuate:

This ensures that LINSTOR will not place resources from a node that you are evacuating onto another node that you plan on evacuating.

If you already ran a node evacuate command that has either completed or still has resources in an “Evacuating” state, you can remove the “Evacuating” state from a node by using the node restore command. This will work if you have not yet run a node delete command.

After restoring the node, you should use the node set-property <node_name> AutoplaceTarget true command, if you previously set the AutoplaceTarget property to “false”. This way, LINSTOR can again place resources onto the node automatically, to fulfill placement count properties that you might have set for resources in your cluster.

Assuming a resource definition named ‘resource1’ which has been placed on some nodes, you can create a snapshot of the resource by entering the following command:

This will create a snapshot on all nodes where the resource is present. LINSTOR will create consistent snapshots, even when the resource is in active use.

Setting the resource definition property AutoSnapshot/RunEvery LINSTOR will automatically create snapshots every X minutes. The optional property AutoSnapshot/Keep can be used to clean up old snapshots which were created automatically. No manually created snapshot will be cleaned up or deleted. If AutoSnapshot/Keep is omitted (or ⇐ 0), LINSTOR will keep the last 10 snapshots by default.

The following steps restore a snapshot to a new resource. This is possible even when the original resource has been removed from the nodes where the snapshots were taken.

首先使用与快照中的卷匹配的卷定义新资源：

At this point, additional configuration can be applied if necessary. Then, when ready, create resources based on the snapshots:

This will place the new resource on all nodes where the snapshot is present. The nodes on which to place the resource can also be selected explicitly; see the help (linstor snapshot resource restore --help).

LINSTOR can roll a resource back to a snapshot state. The resource must not be in use. That is, the resource must not be mounted on any nodes. If the resource is in use, consider whether you can achieve your goal by restoring the snapshot instead.

回滚执行如下：

A resource can only be rolled back to the most recent snapshot. To roll back to an older snapshot, first delete the intermediate snapshots.

可以按如下方式删除现有快照：

In a LINSTOR cluster, a snapshot shipping target is called a remote. Currently, there are two different types of remotes that you can use when shipping snapshots: LINSTOR remotes and S3 remotes. LINSTOR remotes are used to ship snapshots to a different LINSTOR cluster or within the same LINSTOR cluster. S3 remotes are used to ship snapshots to AWS S3, MinIO, or any other service using S3 compatible object storage. A shipped snapshot on a remote is also called a backup.

To ship a snapshot to an S3 remote, that is, to create a backup of a resource on an S3 remote, all you need to do is to specify an S3 remote that the current cluster can reach and then specify the resource that should be shipped. The following command will create a snapshot of a resource, myRsc, and ship it to the given S3 remote, myRemote:

If this isn’t the first time you shipped a backup of this resource (to that remote) and the snapshot of the previous backup hasn’t been deleted yet, an incremental backup will be shipped. To force the creation of a full backup, add the --full argument to the command. Getting a specific node to ship the backup is also possible by using --node myNode, but if the specified node is not available or only has the resource diskless, a different node will be chosen.

You can ship a snapshot between two LINSTOR clusters by specifying a LINSTOR remote. Snapshot shipping to a LINSTOR remote also requires at least one diskful resource on the source side (where you issue the shipping command).

If you want to ship a snapshot inside the same cluster, you just need to create a LINSTOR remote that points to the local controller. To do this if you are logged into your LINSTOR controller, for example, enter the following command:

You can then follow the instructions to ship a snapshot to a LINSTOR remote.

To show which backups exist on a specific S3 remote, use linstor backup list myS3Remote. A resource-name can be added to the command as a filter to only show backups of that specific resource by using the argument --resource myRsc. If you use the --other argument, only entries in the bucket that LINSTOR does not recognize as a backup will be shown. LINSTOR always names backups in a certain way, and if an item in the remote is named according to this schema, it is assumed that it is a backup created by LINSTOR – so this list will show everything else.

To show which backups exist on a LINSTOR remote, use the linstor snapshot list command on your LINSTOR target cluster.

There are several options when it comes to deleting backups:

Maybe the most important task after creating a backup is restoring it. To restore a backup from an S3 remote, you can use the linstor backup restore command.

With this command, you specify the name of the S3 remote to restore a backup from, the name of the target node, and a target resource to restore to on the node. If the resource name does not match an existing resource definition, LINSTOR will create the resource definition and resource.

Additionally, you need to specify the name of the resource on the remote that has the backup. You do this by using either the --resource or --id arguments but not both.

By using the --resource option, you can restore the latest backup of the resource that you specify, for example:

By using the --id option, you can restore the exact backup that you specify, for example, to restore a backup of a resource other than the most recent. To get backup IDs, refer to Listing Backups on a Remote.

When you restore a backup to a resource, LINSTOR will download all the snapshots from the last full backup of the resource, up to the backup that you specified (when using the --id option) or up to the latest backup (when using the --resource option). After downloading these snapshots, LINSTOR restores the snapshots into a new resource. You can skip restoring the snapshots into a new resources by adding the --download-only option to your backup restore command.

LINSTOR can download backups to restore from any cluster, not just the one that uploaded them, provided that the setup is correct. Specifically, the target resource cannot have any existing resources or snapshots, and the storage pool(s) used need to have the same storage providers. If the storage pool(s) on the target node has the exact same name(s) as on the cluster the backup was created on, no extra action is necessary. If the nodes have different storage pool names, then you need to use the --storpool-rename option with your backup restore command. This option expects at least one oldname=newname pair. For every storage pool of the original backup that is not named in that list, LINSTOR assumes that the storage pool name is exactly the same on the target node.

To find out exactly which storage pools you will need rename, and how big the download and the restored resource will be, you can use the command linstor backup info myRemote …​. Similar to the restore command, you need to specify either the --resource or --id option. When used with the backup info command, these options have the same restrictions as when used with the backup restore command. To show how much space will be left over in the local storage pools after a restore, you can add the argument -n myNode. The same as with an actual restore operation, the backup info command assumes that the storage pool names are exactly the same on the target and source nodes. If that is not the case, you can use the --storpool-rename option, just as with the restore command.

You create a backup shipping schedule by using the LINSTOR client schedule create command and defining the frequency of backup shipping using cron syntax. You also need to set options that name the schedule and define various aspects of the backup shipping, such as on-failure actions, the number of local and remote backup copies to keep, and whether to also schedule incremental backup shipping.

At a minimum, the command needs a schedule name and a full backup cron schema to create a backup shipping schedule. An example command would look like this:

You can modify a backup shipping schedule by using the LINSTOR client schedule modify command. The syntax for the command is the same as that for the schedule create command. The name that you specify with the schedule modify command must be an already existing backup schedule. Any options to the command that you do not specify will retain their existing values. If you want to set the keep-local or keep-remote options back to their default values, you can set them to “all”. If you want to set the max-retries option to its default value, you can set it to “forever”.

Your physical storage is not infinite and your remote storage has a cost, so you will likely want to set limits on the number of snapshots and backups you keep.

Both the --keep-remote and --keep-local options deserve special mention as they have implications beyond what might be obvious. Using these options, you specify how many snapshots or full backups should be kept, either on the local source or the remote destination.

You can list your backup shipping schedules by using the linstor schedule list command.

For example:

The LINSTOR client schedule delete command completely deletes a backup shipping schedule LINSTOR object. The command’s only argument is the schedule name that you want to delete. If the deleted schedule is currently creating or shipping a backup, the scheduled shipping process is stopped. Depending on at which point the process stops, a snapshot, or a backup, or both, might not be created and shipped.

This command does not affect previously created snapshots or successfully shipped backups. These will be retained until they are manually deleted.

You can use the LINSTOR client backup schedule enable command to enable a previously created backup shipping schedule. The command has the following syntax:

To disable a previously enabled backup shipping schedule, you use the LINSTOR client backup schedule disable command. The command has the following syntax:

If you include the option specifying either a resource group or resource definition, as described in the backup schedule enable command example above, then you disable the schedule only for that resource group or resource definition.

For example, if you omitted specifying a resource group or resource definition in an earlier backup schedule enable command, LINSTOR would schedule backup shipping for all its deployed resources that can make snapshots. Your disable command would then only affect the resource group or resource definition that you specify with the command. The backup shipping schedule would still apply to any deployed LINSTOR resources besides the specified resource group or resource definition.

The same as for the backup schedule enable command, if you specify neither a resource group nor a resource definition, then LINSTOR disables the backup shipping schedule at the controller level for all deployed LINSTOR resources.

You can use the linstor backup schedule delete command to granularly delete either a specified resource definition or a resource group from a backup shipping schedule, without deleting the schedule itself. This command has the same syntax and arguments as the backup schedule disable command. If you specify neither a resource group nor a resource definition, the backup shipping schedule you specify will be deleted at the controller level.

It might be helpful to think about the backup schedule delete command as a way that you can remove a backup shipping schedule-remote pair from a specified LINSTOR object level, either a resource definition, a resource group, or at the controller level if neither is specified.

The backup schedule delete command does not affect previously created snapshots or successfully shipped backups. These will be retained until they are manually deleted, or until they are removed by the effects of a still applicable keep-local or keep-remote option.

You might want to use this command when you have disabled a backup schedule for multiple LINSTOR object levels and later want to affect a granular change, where a backup schedule enable command might have unintended consequences.

For example, consider a scenario where you have a backup schedule-remote pair that you enabled at a controller level. This controller has a resource group, myresgroup that has several resource definitions, resdef1 through resdef9, under it. For maintenance reasons perhaps, you disable the schedule for two resource definitions, resdef1 and resdef2. You then realize that further maintenance requires that you disable the backup shipping schedule at the resource group level, for your myresgroup resource group.

After completing some maintenance, you are able to enable the backup shipping schedule for resdef3 through resdef9, but you are not yet ready to resume (enable) backup shipping for resdef1 and resdef2. You can enable backup shipping for each resource definition individually, resdef3 through resdef9, or you can use the backup schedule delete command to delete the backup shipping schedule from the resource group, myresgroup. If you use the backup schedule delete command, backups of resdef3 through resdef9 will ship again because the backup shipping schedule is enabled at the controller level, but resdef1 and resdef2 will not ship because the backup shipping schedule is still disabled for them at the resource definition level.

When you complete your maintenance and are again ready to ship backups for resdef1 and resdef2, you can delete the backup shipping schedule for those two resource definitions to return to your starting state: backup shipping scheduled for all LINSTOR deployed resources at the controller level. To understand this it might be helpful to refer to the decision tree diagram for how LINSTOR decides whether or not to ship a backup in the How the LINSTOR Controller Determines Scheduled Backup Shipping subsection.

You can list backup schedules by resource, using the LINSTOR client schedule list-by-resource command. This command will show LINSTOR resources and how any backup shipping schedules apply and to which remotes they are being shipped. If resources are not being shipped then the command will show:

If resources have schedule-remote-pairs and are being shipped, the command output will show when the last backup was shipped and when the next backup is scheduled to ship. It will also show whether the next and last backup shipments were full or incremental backups. Finally, the command will show when the next planned incremental (if any) and full backup shipping will occur.

You can use the --active-only flag with the schedule list-by-resource command to filter out all resources that are not being shipped.

To determine if the LINSTOR Controller will ship a deployed LINSTOR resource with a certain backup schedule for a given remote destination, the LINSTOR Controller uses the following logic:

As the diagram shows, enabled or disabled backup shipping schedules have effect in the following order:

A backup shipping schedule-remote pair that is enabled or disabled at a preceding level will override the enabled or disabled status for the same schedule-remote pair at a later level.

To determine how a LINSTOR resource will be affected by scheduled backup shipping, you can use the LINSTOR client schedule list-by-resource-details command for a specified LINSTOR resource.

The command will output a table that shows on what LINSTOR object level a backup shipping schedule is either not set (empty cell), enabled, or disabled.

By using this command, you can determine on which level you need to make a change to enable, disable, or delete scheduled backup shipping for a resource.

Example output could look like this:

The LINSTOR resource object is an exception to the general syntax for setting DRBD options for LINSTOR objects. With the LINSTOR resource object, you can use the drbd-peer-options to set DRBD options at the connection level between the two nodes that you specify. Specifying drbd-peer-options for a LINSTOR resource object between two nodes is equivalent to using the`linstor resource-connection drbd-peer-options` for a resource between two nodes.

For example, to set the DRBD maximum buffer size to 8192 at a connection level, for a resource named backups, between two nodes, node-0 and node-1, enter:

The command above is equivalent to the following:

Indeed, when using the linstor --curl command to examine the two commands actions on the LINSTOR REST API, the output is identical:

The connection section of the LINSTOR-generated resource file backups.res on node-0 will look something like this:

You can use the drbd-peer-options argument to set DRBD options at a connection level, between two nodes, for example:

The preceding command would set the DRBD ping-timeout option to 29.9 seconds at a connection level between two nodes, node-0 and node-1.

You can verify a LINSTOR object’s set properties by using the list-properties command, for example:

To remove a previously set DRBD option, prefix the option name with unset-. For example:

The same syntax applies to any drbd-peer-options set either on a LINSTOR resource, resource connection, or node connection. For example:

Removing a DRBD option or DRBD peer option will return the option to its default value. Refer to the linstor <LINSTOR_object> drbd-options --help (or drbd-peer-options --help) command output for the default values of options. You can also refer to the drbd.conf-9.0 man page to get information about DRBD options. === Over Provisioning Storage in LINSTOR

Since LINSTOR server version 1.26.0, it is possible to control how LINSTOR limits over provisioning storage by using three LINSTOR object properties: MaxOversubscriptionRatio, MaxFreeCapacityOversubscriptionRatio, and MaxTotalCapacityOversubscriptionRatio. You can set these properties on either a LINSTOR storage pool or a LINSTOR controller object. Setting these properties at the controller level will affect all LINSTOR storage pools in the cluster, unless the same property is also set on a specific storage pool. In this case, the property value set on the storage pool will take precedence over the property value set on the controller. By setting values for over subscription ratios, you can control how LINSTOR limits the over provisioning that you can do with your storage pools. This might be useful in cases where you do not want over provisioning of storage to reach a level that you cannot account for physically, if you needed to. The following subsections discuss the different over provisioning limiting properties. ==== Configuring a Maximum Free Capacity Over Provisioning Ratio

In LINSTOR server versions before 1.26.0, configuring over provisioning of a storage pool backed by a thin-provisioned LVM volume or ZFS zpool was based on the amount of free space left in the storage pool. In LINSTOR server versions from 1.26.0, this method of over provisioning is still possible. You do this by setting a value for the MaxFreeCapacityOversubscriptionRatio property on either a LINSTOR storage pool or the LINSTOR controller. When you configure a value for this ratio, the remaining free space in a storage pool is multiplied by the ratio value and this becomes the maximum allowed size for a new volume that you can provision from the storage pool.

By default, the MaxFreeCapacityOversubscriptionRatio has a value of 20.

To configure a different value for the MaxFreeCapacityOversubscriptionRatio property on a LINSTOR storage pool named my_thin_pool on three LINSTOR satellite nodes in a cluster, named node-0 through node-2, enter the following command:

If you had already deployed some storage resources and the storage pool had 10GiB free capacity remaining (shown by entering linstor storage-pool list), then when provisioning a new storage resource from the storage pool, you could at most provision a 100GiB volume.

Before spawning new LINSTOR resources (and volumes) from a resource group, you can list the maximum volume size that LINSTOR will let you provision from the resource group. To do this, enter a resource-group query-size-info command. For example, to list information about the DfltRscGrp resource group, enter the following command:

Output from the command will show a table of values for the specified resource group:

Since LINSTOR server version 1.26.0, there is an additional way that you can limit over provisioning storage. By setting a value for the MaxTotalCapacityOversubscriptionRatio on either a LINSTOR storage pool or LINSTOR controller, LINSTOR will limit the maximum size of a new volume that you can deploy from a storage pool, based on the total capacity of the storage pool.

This is a more relaxed way to limit over provisioning, compared to limiting the maximum volume size based on the amount of free space that remains in a storage pool. As you provision and use more storage in a storage pool, the free space decreases. However, the total capacity of a storage pool does not change, unless you add more backing storage to the storage pool.

By default, the MaxTotalCapacityOversubscriptionRatio has a value of 20.

To configure a different value for the MaxTotalCapacityOversubscriptionRatio property on a LINSTOR storage pool named my_thin_pool on three LINSTOR satellite nodes in a cluster, named node-0 through node-2, enter the following command:

If you had a storage pool backed by a thin-provisioned logical volume that had a total capacity of 10GiB, then each new storage resource that you created from the storage pool could have a maximum size of 40GiB, regardless of how much free space remained in the storage pool.

If you set the MaxOversubscriptionRatio property on either a LINSTOR storage pool or LINSTOR controller object, it can act as the value for both the MaxFreeCapacityOversubscriptionRatio and MaxTotalCapacityOversubscriptionRatio properties. However, the MaxOversubscriptionRatio property will only act as the value for either of the other two over subscription properties if the other property is not set. By default, the value for the MaxOversubscriptionRatio property is 20, the same as the default values for the other two over subscription properties.

To configure a different value for the MaxOversubscriptionRatio property on a LINSTOR controller, for example, enter the following command:

By setting the property value to 5 in the command example, you would be able to over provision a 10GiB storage pool, for example, by up to 40GiB.

As previously mentioned, the default value for each of the three over provisioning limiting LINSTOR properties is 20. If you set the same property on a storage pool and the controller, then the value of the property set on the storage pool takes precedence for that storage pool.

If either MaxTotalCapacityOversubscriptionRatio or MaxFreeCapacityOversubscriptionRatio is not set, the value of MaxOversubscriptionRatio is used as the value for the unset property or properties.

For example, consider the following commands:

Because you set the MaxTotalCapacityOversubscriptionRatio property to 3, LINSTOR will use that value for the property, rather than the value that you set for the MaxOversubscriptionRatio property. Because you did not set the MaxFreeCapacityOversubscriptionRatio property, LINSTOR will use the value that you set for the MaxOversubscriptionRatio property, 4, for the MaxFreeCapacityOversubscriptionRatio property.

When determining resource placement, that is, when LINSTOR decides whether or not a resource can be placed in a storage pool, LINSTOR will compare two values: the ratio set by the MaxTotalCapacityOversubscriptionRatio property multiplied by the total capacity of the storage pool, and the ratio set by the MaxFreeCapacityOversubscriptionRatio property multiplied by the available free space of the storage pool. LINSTOR will use the lower of the two calculated values to determine whether the storage pool has enough space to place the resource.

Consider the case of values set by earlier commands and given a total capacity of 10GiB for the storage pool. Also assume that there are no deployed LINSTOR resources associated with the storage pool and that the available free space of the storage pool is also 10GiB.

In this example, MaxTotalCapacityOversubscriptionRatio, 3, multiplied by the total capacity, 10GiB, is less than the MaxFreeCapacityOversubscriptionRatio, 4, multiplied by the free capacity, 10GiB. Remember here that because the MaxFreeCapacityOversubscriptionRatio is unset, LINSTOR uses the MaxOversubscriptionRatio value.

Therefore, when deploying a new resource backed by the storage pool, LINSTOR would use the total capacity calculation, rather than the free capacity calculation, to limit over provisioning the storage pool and determine whether a new resource can be placed. However, as you deploy more storage resources and they fill up with data, the available free space in the storage pool will decrease. For this reason, it might not always be the case that LINSTOR will use the total capacity calculation to limit over provisioning the storage pool, even though its ratio is less that the free capacity ratio.

To move a resource between nodes without reducing redundancy at any point, you can use the LINSTOR disk migration feature. First, create a diskless resource on the target node, and then toggle the resource to diskful by using the --migrate-from option. This LINSTOR client operation will wait until the data has been synced to the new disk and then it will remove the source disk.

For example, to migrate a resource backups from ‘alpha’ to ‘bravo’:

LINSTOR can also be configured to automatically enable the above mentioned Proxy connection between two nodes. For this automation, LINSTOR first needs to know on which site each node is.

As the Site property might also be used for other site-based decisions in future features, the DrbdProxy/AutoEnable also has to be set to true:

This property can also be set on node, resource-definition, resource and resource-connection level (from left to right in increasing priority, whereas the controller is the leftmost, that is, the least prioritized level).

Once this initialization steps are completed, every newly created resource will automatically check if it has to enable DRBD Proxy to any of its peer-resources.

A sample PostgreSQL linstor.toml looks like this:

A sample MariaDB linstor.toml looks like this:

etcd is a distributed key-value store that makes it easy to keep your LINSTOR database distributed in a HA-setup. The etcd driver is already included in the LINSTOR-controller package and only needs to be configured in the linstor.toml.

More information about how to install and configure etcd can be found here: etcd docs

And here is a sample [db] section from the linstor.toml:

To make LINSTOR’s administrative tasks more accessible and also available for web-frontends a REST API has been created. The REST API is embedded in the LINSTOR controller and since LINSTOR 0.9.13 configured through the linstor.toml configuration file.

If you want to use the REST API the current documentation can be found on the following link: https://app.swaggerhub.com/apis-docs/Linstor/Linstor/

The HTTP REST API can also run secured by HTTPS and is highly recommended if you use any features that require authorization. To do so you have to create a Java keystore file with a valid certificate that will be used to encrypt all HTTPS traffic.

Here is a simple example on how you can create a self signed certificate with the keytool that is included in the Java Runtime:

keytool will ask for a password to secure the generated keystore file and is needed for the LINSTOR Controller configuration. In your linstor.toml file you have to add the following section:

Now (re)start the linstor-controller and the HTTPS REST API should be available on port 3371.

More information about how to import other certificates can be found here: https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html

The LINSTOR-Controller also provides a /health HTTP path that will simply return HTTP-Status 200 if the controller can access its database and all services are up and running. Otherwise it will return HTTP error status code 500 Internal Server Error.

After configuring the LINSTOR Controller to authenticate users through LDAP (or LDAPS), and the LINSTOR REST API HTTPS, you will need to enter LINSTOR commands as follows:

If you have configured LDAP authentication without also configuring LINSTOR REST API HTTPS, you will need to explicitly enable password authentication over HTTP, by using the --allow-insecure-path flag with your linstor commands. This is not recommended outside of a secured and isolated LAN, as you will be sending credentials in plain text.

The LINSTOR Controller will prompt you for the user’s password, in each of the above examples. You can optionally use the --password argument to supply the user’s password on the command line, with all the warnings of caution that would go along with doing so.

LINSTOR automatically configures quorum policies on resources when quorum is achievable. This means, whenever you have at least two diskful and one or more diskless resource assignments, or three or more diskful resource assignments, LINSTOR will enable quorum policies for your resources automatically.

Inversely, LINSTOR will automatically disable quorum policies whenever there are less than the minimum required resource assignments to achieve quorum.

This is controlled through the, DrbdOptions/auto-quorum, property which can be applied to the linstor-controller, resource-group, and resource-definition LINSTOR objects. Accepted values for the DrbdOptions/auto-quorum property are disabled, suspend-io, and io-error.

Setting the DrbdOptions/auto-quorum property to disabled will allow you to manually, or more granularly, control the quorum policies of your resources should you want to.

For example, to manually set the quorum policies of a resource group named my_ssd_group, you would use the following commands:

You might want to disable DRBD’s quorum features completely. To do that, you would need to first disable DrbdOptions/auto-quorum on the appropriate LINSTOR object, and then set the DRBD quorum features accordingly. For example, use the following commands to disable quorum entirely on the my_ssd_group resource group:

If a satellite is offline for a prolonged period of time, LINSTOR can be configured to declare that node as evicted. This triggers an automated reassignment of the affected DRBD resources to other nodes to ensure a minimum replica count is kept.

This feature uses the following properties to adapt the behavior.

After the LINSTOR controller loses the connection to a satellite, aside from trying to reconnect, it starts a timer for that satellite. As soon as that timer exceeds DrbdOptions/AutoEvictAfterTime and all of the DRBD connections to the DRBD resources on that satellite are broken, the controller will check whether or not DrbdOptions/AutoEvictMaxDisconnectedNodes has been met. If it has not, and DrbdOptions/AutoEvictAllowEviction is true for the node in question, the satellite will be marked as EVICTED. At the same time, the controller will check for every DRBD-resource whether the number of resources is still above DrbdOptions/AutoEvictMinReplicaCount. If it is, the resource in question will be marked as DELETED. If it isn’t, an auto-place with the settings from the corresponding resource-group will be started. Should the auto-place fail, the controller will try again later when changes that might allow a different result, such as adding a new node, have happened. Resources where an auto-place is necessary will only be marked as DELETED if the corresponding auto-place was successful.

The evicted satellite itself will not be able to reestablish connection with the controller. Even if the node is up and running, a manual reconnect will fail. It is also not possible to delete the satellite, even if it is working as it should be. The satellite can, however, be restored. This will remove the EVICTED-flag from the satellite and allow you to use it again. Previously configured network interfaces, storage pools, properties and similar entities, non-DRBD-related resources, and resources that LINSTOR could not autoplace somewhere else will still be on the satellite. To restore a satellite, use the following command:

Should you want to instead throw everything that once was on that node, including the node itself, away, you need to use the node lost command instead.

You can set the LINSTOR auto-diskful and auto-diskful-allow-cleanup properties for various LINSTOR objects, for example, a resource definition, to have LINSTOR help automatically make a Diskless node Diskful and perform appropriate cleanup actions afterwards.

This is useful when a Diskless node has been in a Primary state for a DRBD resource for more than a specified number of minutes. This could happen in cases where you integrate LINSTOR managed storage with other orchestrating and scheduling platforms, such as OpenStack, OpenNebula, and others. On some platforms that you integrate LINSTOR with, you might not have a way to influence where in your cluster a storage volume will be used.

The auto-diskful options give you a way to use LINSTOR to sensibly delegate the roles of your storage nodes in response to an integrated platform’s actions that are beyond your control.

If a device is throwing I/O errors, for example, due to physical failure, DRBD detects this state and automatically detaches from the local disk. All read and write requests are forwarded to a still healthy peer, allowing the application to continue without interruption.

This automatic detaching causes new event entries in the drbdsetup events2 stream, changing the state of a DRBD volume from UpToDate to Failed and finally to Diskless. LINSTOR detects these state changes, and automatically sets the DrbdOptions/SkipDisk property to True on the given resource. This property causes the LINSTOR satellite service running on the node with the device throwing I/O errors to attach a --skip-disk to all drbdadm adjust commands.

If this property is set, the linstor resource list command also shows it accordingly:

The (R) in this case already states that the property is set on the resource. The indicator would change to (R, N) if the property DrbdOptions/SkipDisk were to be set on the resource and on node levels.

Although this property is automatically enabled by LINSTOR, after you resolve the cause of the I/O errors, you need to manually remove the property to get back to a healthy UpToDate state:

To set the property at the node level, for example, enter the following command:

To verify that the property is set on a specified LINSTOR object, you can use the list-properties subcommand.

Output from the command will show a table of set properties and their values for the specified LINSTOR object.

To unset the property, enter the same command but do not specify a storage pool name, as in the following example:

These LINSTOR properties can be set by using the set-property command and can be set on the following objects: volume, storage pool, resource, controller, or node. You can also set these QoS properties on resource groups, volume groups, resource definitions, or volume definitions. When you set a QoS property on a group or definition, resources created from the group or definition will inherit the QoS settings.

The following example shows creating a resource group, then creating a volume group, then applying QoS settings to the volume group, and then spawning resources from the resource group. A verification command will show that the spawned resources inherit the QoS settings. The example uses an assumed previously created LINSTOR storage pool named pool1. You will need to replace this name with a storage pool name that exists in your environment.

To verify that the spawned resources inherited the QoS setting, you can show the contents of the corresponding sysfs file, on a node that contributes storage to the storage pool.

A single LINSTOR volume can be composed of multiple DRBD devices. For example, DRBD with external metadata will have three backing devices: a data (storage) device, a metadata device, and the composite DRBD device (volume) provided to LINSTOR. If the data and metadata devices correspond to different backing disks, then if you set a sysfs property for such a LINSTOR volume, only the local data (storage) backing device will receive the property value in the corresponding /sys/fs/cgroup/blkio/ file. Neither the device backing DRBD’s metadata, nor the composite backing device provided to LINSTOR would receive the value. However, when DRBD’s data and its metadata share the same backing disk, QoS settings will affect the performance of both data and metadata operations.

In case a LINSTOR resource definition has an nvme-target and an nvme-initiator resource, both data (storage) backing devices of each node will receive the sysfs property value. In case of the target, the data backing device will be the volume of either LVM or ZFS, whereas in case of the initiator, the data backing device will be the connected nvme-device, regardless of which other LINSTOR layers, such as LUKS, NVMe, DRBD, and others (see Using LINSTOR Without DRBD), are above that.

A quick way to list available commands on the command line is to enter linstor.

Further information about subcommands, for example, listing nodes) can be retrieved in two ways:

当LINSTOR以交互模式（LINSTOR interactive）执行时，使用 help 子命令尤其有用。

One of the most helpful features of LINSTOR is its rich tab-completion, which can be used to complete nearly every object that LINSTOR knows about, for example, node names, IP addresses, resource names, and others. The following examples show some possible completions, and their results:

If tab-completion does not work upon installing the LINSTOR client, try to source the appropriate file:

For Z shell users, the linstor-client command can generate a Z shell compilation file, that has basic support for command and argument completion.

If something goes wrong and you need help finding the cause of the issue, you can enter the following command:

The command above will create a new sos-report in the /var/log/linstor/controller directory on the controller node. Alternatively you can enter the following command:

This command will create a new SOS report and download that report to the current working directory on your local machine.

This SOS report contains logs and useful debug-information from several sources (LINSTOR logs, dmesg, versions of external tools used by LINSTOR, ip a, database dump and many more). These information are stored for each node in plain text in the resulting .tar.gz file.

For help from the community, subscribe to the DRBD user mailing list located here: https://lists.linbit.com/listinfo/drbd-user

To file bug or feature request, check out the LINBIT GitHub page https://github.com/linbit

Alternatively, if you need to purchase remote installation services, 24×7 support, access to certified repositories, or feature development, contact us: +1-877-454-6248 (1-877-4LINBIT) , International: +43-1-8178292-0 | [email protected]

To deploy the Operator, create a kustomization.yaml file. This will declare your pull secret for drbd.io and allow you to pull in the Operator deployment. The Operator will be deployed in a new namespace linbit-sds. Make sure to replace MY_LINBIT_USER and MY_LINBIT_PASSWORD with your own credentials. You can find the latest releases on charts.linstor.io.

Then, apply the kustomization.yaml file, by using kubectl command, and wait for the Operator to start:

The Operator is now ready to deploy LINBIT SDS for Kubernetes.

Deploying LINBIT SDS for Kubernetes with the Operator v2 is as simple as creating a new LinstorCluster resource and waiting for the Operator to complete the setup:

Output should eventually show that the wait-for condition has been met and the LINBIT SDS pods are up and running.

To create the LINSTOR Operator v2 by Using Helm, first enter the following commands to add the linstor Helm chart repository:

Next, enter the following command to install the LINSTOR Operator v2:

After output from the command shows that the Operator v2 was installed, you can use Helm to deploy LINBIT SDS by installing the linbit-sds chart:

Output from this final helm install command should show a success message.

By default, LINBIT SDS for Kubernetes does not configure any storage. To add storage, you can configure a LinstorSatelliteConfiguration, which the Operator uses to configure one or more satellites.

The following example creates a simple FILE_THIN pool and it does not require any additional set up on the host:

Other types of storage pools can be configured as well. Refer to the examples upstream.

By configuring key and certificate based encryption, you can make communication between certain LINSTOR components, for example, between LINSTOR satellite nodes and a LINSTOR controller node, or between the LINSTOR client and the LINSTOR API, more secure.

After deploying LINBIT SDS for Kubernetes, you can continue with the [s-kubernetes-basic-configuration-and-deployment], Configuring the DRBD Module Loader in Operator v2 Deployments, Using the Host Network for DRBD Replication in Operator v2 Deployments sections in this chapter, or refer to the available tutorials in the upstream Piraeus project. === Deploying LINSTOR Operator v1

The Operator v1 is installed using a Helm v3 chart as follows:

The LINSTOR controller can use the Kubernetes API directly to persist its cluster state. To enable this back end, use the following override file during the chart installation:

You can use the pv-hostpath Helm templates to create hostPath persistent volumes. Create as many PVs as needed to satisfy your configured etcd replicas (default 1).

Create the hostPath persistent volumes, substituting cluster node names accordingly in the nodes= option:

By default, a PV is created on every control-plane node. You can manually select the storage nodes by passing --set "nodes={<NODE0>,<NODE1>,<NODE2>}" to the install command. NOTE: The correct value to reference the node is the value of the kubernetes.io/hostname label. You can list the value for all nodes by running kubectl get nodes -o custom-columns="Name:{.metadata.name},NodeName:{.metadata.labels['kubernetes\.io/hostname']}" ==== Using an Existing Database

LINSTOR can connect to an existing PostgreSQL, MariaDB or etcd database. For instance, for a PostgreSQL instance with the following configuration:

The Helm chart can be configured to use this database rather than deploying an etcd cluster, by adding the following to the Helm install command:

The LINSTOR Operator v1 can automate some basic storage set up for LINSTOR.

The default communication between LINSTOR components is not secured by TLS. If this is needed for your setup, choose one of three methods: // “cert-manager” is a product name so keep the original case

All the below examples use the following sp-values.yaml file. Feel free to adjust this for your uses and environment. See [Configuring storage pool creation] for further details.

Install with LINSTOR storage-pools defined at install through sp-values.yaml, persistent hostPath volumes, three etcd replicas, and by compiling the DRBD kernel modules for the host kernels.

This should be adequate for most basic deployments. Note that this deployment is not using the pre-compiled DRBD kernel modules just to make this command more portable. Using the pre-compiled binaries will make for a much faster install and deployment. Using the Compile option would not be suggested for use in a large Kubernetes clusters.

Install with LINSTOR storage-pools defined at install through sp-values.yaml, use an already created PostgreSQL DB (preferably clustered), rather than etcd, and use already compiled kernel modules for DRBD.

The PostgreSQL database in this particular example is reachable through a service endpoint named postgres. PostgreSQL itself is configured with POSTGRES_DB=postgresdb, POSTGRES_USER=postgresadmin, and POSTGRES_PASSWORD=admin123

To protect the storage infrastructure of the cluster from accidentally deleting vital components, it is necessary to perform some manual steps before deleting a Helm deployment.

The Helm charts provide a set of further customization options for advanced use cases.

To create a high-availability deployment of all components within a LINSTOR Operator v1 deployment, consult the upstream guide The default values are chosen so that scaling the components to multiple replicas ensures that the replicas are placed on different nodes. This ensures that a single node failures will not interrupt the service. NOTE: If you have deployed LINBIT SDS in Kubernetes by using the LINSTOR Operator v2, high availability is built into the deployment by default. ===== Fast Workload Failover Using the High Availability Controller

When node failures occur, Kubernetes is very conservative in rescheduling stateful workloads. This means it can take more than 15 minutes for Pods to be moved from unreachable nodes. With the information available to DRBD and LINSTOR, this process can be sped up significantly.

The LINSTOR High Availability Controller (HA Controller) speeds up the failover process for stateful workloads using LINSTOR for storage. It monitors and manages any Pod that is attached to at least one DRBD resource.

For the HA Controller to work properly, you need quorum, that is at least three replicas (or two replicas + one diskless tiebreaker). If using lower replica counts, attached Pods will be ignored and are not eligible for faster failover.

The HA Controller is packaged as a Helm chart, and can be deployed using:

If you are using the HA Controller in your cluster you can set additional parameters in all StorageClasses. These parameters ensure that the volume is not accidentally remounted as read-only, leading to degraded Pods.

To exempt a Pod from management by the HA Controller, add the following annotation to the Pod:

To create a backup of the etcd database (in LINSTOR Operator v1 deployments) and store it on your control host, enter the following commands:

These commands will create a file save.db on the machine you are running kubectl from. === Deploying with an External LINSTOR Controller

The Operator can configure the satellites and CSI plugin to use an existing LINSTOR setup. This can be useful in cases where the storage infrastructure is separate from the Kubernetes cluster. Volumes can be provisioned in diskless mode on the Kubernetes nodes while the storage nodes will provide the backing disk storage. ==== Operator v2 Deployment with an External LINSTOR Controller ===== Configuring the LinstorCluster Resource

To use an externally managed LINSTOR cluster, specify the URL of the LINSTOR controller in the LinstorCluster resource in a YAML configuration and apply it to your deployment. In the following example, the LINSTOR controller is reachable at http://linstor-controller.example.com:3370.

Normally the pod network is not reachable from outside the Kubernetes cluster. In this case the external LINSTOR controller would not be able to communicate with the satellites in the Kubernetes cluster. For this reason, you need to configure your satellites to use host networking.

To use host networking, deploy a LinstorSatelliteConfiguration resource by applying the following YAML configuration to your deployment:

To skip the creation of a LINSTOR controller deployment and configure the other components to use your existing LINSTOR controller, use the following options when running helm install:

After all pods are ready, you should see the Kubernetes cluster nodes as satellites in your LINSTOR setup.

Create a storage class referencing an existing storage pool on your storage nodes.

You can provision new volumes by creating PVCs using your storage class. The volumes will first be placed only on nodes with the given storage pool, that is, your storage infrastructure. Once you want to use the volume in a pod, LINSTOR CSI will create a diskless resource on the Kubernetes node and attach over the network to the diskful resource. === 在Kubernetes中与LINSTOR互动

The controller pod includes a LINSTOR Client, making it easy to interact directly with LINSTOR. For example:

To simplify entering LINSTOR client commands within a Kubernetes deployment, you can use the kubectl-linstor utility. This utility is available from the upstream Piraeus datastore project. To download it, enter the following commands on your Kubernetes control plane node:

To install the utility, first extract it and then move the extracted executable file to a directory in your $PATH, for example, /usr/bin. Then you can use kubectl-linstor to get access to the complete LINSTOR CLI.

It also expands references to PVCs to the matching LINSTOR resource.

It also expands references of the form pod:[<namespace>/]<podname> into a list resources in use by the pod. This should only be necessary for investigating problems and accessing advanced functionality. Regular operation such as creating volumes should be achieved through the Kubernetes integration. === Getting Started with LINBIT SDS Storage in Kubernetes

Once all linstor-csi Pods are up and running, you can provision volumes using the usual Kubernetes workflows.

Configuring the behavior and properties of LINSTOR volumes deployed through Kubernetes is accomplished using Kubernetes StorageClass objects.

Here below is the simplest practical StorageClass that can be used to deploy volumes:

You can create the storage class with the following command:

Now that your storage class is created, you can now create a persistent volume claim (PVC) which can be used to provision volumes known both to Kubernetes and LINSTOR:

You can create the PersistentVolumeClaim with the following command:

This will create a PersistentVolumeClaim, but no volume will be created just yet. The storage class we used specified volumeBindingMode: WaitForFirstConsumer, which means that the volume is only created once a workload starts using it. This ensures that the volume is placed on the same node as the workload.

For our example, we create a simple Pod, which mounts or volume by referencing the PersistentVolumeClaim. .my-first-linstor-volume-pod.yaml

You can create the Pod with the following command:

Running kubectl describe pod fedora can be used to confirm that Pod scheduling and volume attachment succeeded. Examining the PersistentVolumeClaim, we can see that it is now bound to a volume.

To remove a volume, verify that no pod is using it and then delete the PersistentVolumeClaim using the kubectl command. For example, to remove the volume that we just made, run the following two commands, noting that the Pod must be unscheduled before the PersistentVolumeClaim will be removed:

The following storage class contains all currently available parameters to configure the provisioned storage.

As shown in the Available Parameters in a Storage Class section, you can set DRBD options within a storage class configuration. Because of the one-to-one correspondence between a StorageClass Kubernetes object and its named LINSTOR resource group (resourceGroup parameter), setting DRBD options within a storage class configuration is similar to setting DRBD options on the LINSTOR resource group.

There are some differences. If you set DRBD options within a storage class configuration, these options will only affect new volumes that are created from the storage class. The options will not affect existing volumes. Furthermore, because you cannot just update the storage class, you will have to delete it and create it again if you add DRBD options to the storage class’s configuration. If you set DRBD options on the LINSTOR resource group object (linstor resource-group set-property <rg-name> DrbdOptions/<option-name>), changes will affect future and existing volumes belonging to the resource group.

Because of the potential pitfalls here, it is simpler to set DRBD options on the LINSTOR controller object. DRBD options set on the controller will apply to all resources groups, resources, and volumes, unless you have also set the same options on any of those LINSTOR objects. Refer to the LINSTOR Object Hierarchy section for more details about LINSTOR object hierarchy.

You can list the properties, including DRBD options, that you can set on the LINSTOR controller object by entering the following command:

The csi.storage.k8s.io/fstype parameter sets the file system type to create for volumeMode: FileSystem PVCs. Currently supported are: * ext4 (default) * xfs ==== autoPlace

autoPlace is an integer that determines the amount of replicas a volume of this StorageClass will have. For example, autoPlace: "3" will produce volumes with three-way replication. If neither autoPlace nor nodeList are set, volumes will be automatically placed on one node. IMPORTANT: If you use this option, you must not use nodeList. IMPORTANT: You have to use quotes, otherwise Kubernetes will complain about a malformed StorageClass. TIP: This option (and all options which affect auto-placement behavior) modifies the number of LINSTOR nodes on which the underlying storage for volumes will be provisioned and is orthogonal to which kubelets those volumes will be accessible from.

placementCount is an alias for autoPlace

The LINSTOR Resource Group (RG) to associate with this StorageClass. If not set, a new RG will be created for each new PVC. ==== storagePool

storagePool is the name of the LINSTOR storage pool that will be used to provide storage to the newly-created volumes. CAUTION: Only nodes configured with this same storage pool with be considered for auto-placement. Likewise, for StorageClasses using nodeList all nodes specified in that list must have this storage pool configured on them. ==== disklessStoragePool

disklessStoragePool is an optional parameter that only affects LINSTOR volumes that are assigned as “diskless” to kubelets, that is, as clients. If you have a custom diskless storage pool defined in LINSTOR, you will specify that here.

A comma-separated list of layers to use for the created volumes. The available layers and their order are described towards the end of this section. Defaults to drbd,storage ==== placementPolicy

Select from one of the available volume schedulers: * AutoPlaceTopology, the default: Use topology information from Kubernetes together with user provided constraints (see [s-kubernetes-replicasonsame] and [s-kubernetes-replicasondifferent]). * AutoPlace Use the LINSTOR auto-placement feature, influenced by [s-kubernetes-replicasonsame] and [s-kubernetes-replicasondifferent] * FollowTopology: Use CSI Topology information to place at least one volume in each “preferred” zone. Only usable if CSI Topology is enabled. * Manual: Use only the nodes listed in nodeList and clientList. * Balanced: EXPERIMENTAL Place volumes across failure domains, using the least used storage pool on each selected node. ==== allowRemoteVolumeAccess

Control on which nodes a volume is accessible. The value for this option can take two different forms:

encryption is an optional parameter that determines whether to encrypt volumes. LINSTOR must be configured for encryption for this to work properly. ==== nodeList

nodeList is a list of nodes for volumes to be assigned to. This will assign the volume to each node and it will be replicated among all of them. This can also be used to select a single node by hostname, but it’s more flexible to use replicasOnSame to select a single node. IMPORTANT: If you use this option, you must not use autoPlace. TIP: This option determines on which LINSTOR nodes the underlying storage for volumes will be provisioned and is orthogonal from which kubelets these volumes will be accessible.

clientList is a list of nodes for diskless volumes to be assigned to. Use in conjunction with [s-kubernetes-nodelist]. ==== replicasOnSame

replicasOnDifferent takes a list of properties to consider, same as replicasOnSame. There are two modes of using replicasOnDifferent: * Preventing volume placement on specific nodes: + If a value is given for the property, the nodes which have that property-value pair assigned will be considered last. + Example: replicasOnDifferent: "no-csi-volumes=true" will place no volume on any node with property no-csi-volumes=true unless there are not enough other nodes to fulfill the autoPlace setting. * Distribute volumes across nodes with different values for the same key: + If no property value is given, LINSTOR will place the volumes across nodes with different values for that property if possible. + Example: Assuming there are 4 nodes, node-a1 and node-a2 are configured with zone=a. node-b1 and node-b2 are configured with zone=b. Using a StorageClass with autoPlace: "2" and replicasOnDifferent: "zone", LINSTOR will create one replica on either node-a1 or node-a2 and one replica on either node-b1 or node-b2.

Create a diskless resource on all nodes that were not assigned a diskful resource.

Do not place the resource on a node which has a resource with a name matching the regular expression. ==== fsOpts fsOpts is an optional parameter that passes options to the volume’s file system at creation time. IMPORTANT: These values are specific to your chosen file system. ==== mountOpts mountOpts is an optional parameter that passes options to the volume’s file system at mount time.

Extra arguments to pass to xfs_io, which gets called before right before first use of the volume. ==== property.linstor.csi.linbit.com/*

Parameters starting with property.linstor.csi.linbit.com/ are translated to LINSTOR properties that are set on the Resource Group associated with the StorageClass.

For example, to set DrbdOptions/auto-quorum to disabled, use:

The full list of options is available here

Snapshots create a copy of the volume content at a particular point in time. This copy remains untouched when you make modifications to the volume content. This, for example, enables you to create backups of your data before performing modifications or deletions on your data. Because a backup is useless unless you have a way to restore it, this section describes how to create a snapshot, and how to restore it, for example, in the case of accidental deletion of your data. The next subsection contains instructions around snapshots within Operator v2 deployments. If you have deployed LINBIT SDS in Kubernetes by using Operator v1, skip ahead to the [s-kubernetes-add-snaphot-support-v1] subsection. ==== Working With Snapshots

Before you can add snapshot support within a LINBIT SDS deployment, you need to meet the following environment prerequisites:

LINSTOR can store snapshots on S3 compatible storage for disaster recovery. This is integrated in Kubernetes using a special VolumeSnapshotClass:

Refer to the instructions in the Shipping Snapshots section for the exact meaning of the snap.linstor.csi.linbit.com/ parameters. The credentials used to log in are stored in a separate secret, as show in the example above.

Referencing the above storage class when creating snapshots causes the snapshots to be automatically uploaded to the configured S3 storage.

The following example show common scenarios where you want to optimize volume accessibility and locality. It also includes examples of how to spread volume replicas across zones in a cluster.

In some circumstances it might be necessary to disable the DRBD module loader entirely. For example, if you are using an immutable operating system, and DRBD and other modules are loaded as part of the host configuration.

To disable the DRBD module loader completely, apply the following YAML configuration to your deployment:

By default, the Operator will try to find a DRBD module loader that matches the host operating system. The Operator determines the host distribution by inspecting the .status.nodeInfo.osImage field of the Kubernetes Node resource. A user-defined image can be used if the automatic mapping does not succeed or if you have different module loading requirements.

The following YAML configuration overrides the chosen DRBD module loader image with a user-defined image example.com/drbd-loader:v9:

drbd.io, available to LINBIT customers only, maintains the following module loader container images:

If you need to create a module loader image for your own distribution, you can refer to the container source files which are available in the upstream Piraeus project.

By default, the DRBD module loader will try to build the kernel module from source. The module loader can also be configured to load the module from a DEB or RPM package included in the image, or skip loading DRBD entirely.

To change the behavior of the DRBD module loader, set the LB_HOW environment variable to an appropriate value shown in the following table:

After setting the LB_HOW environment variable, apply the following YAML configuration to your deployment. Based on the name within the metadata section, the example below would be used with an LB_HOW environment variable that was set to deps_only.

Switching from the default container network to the host network for DRBD replication is possible at any time. Existing DRBD resources will then be reconfigured to use the host network interface.

To configure the host network for the LINSTOR satellite, apply the following YAML configuration to your deployment:

After the satellite pods are recreated, they will use the host network. Any existing DRBD resources are reconfigured to use a new IP address on the host network rather than an IP address on the container network.

Switching back from host network to container network involves manually resetting the configured peer addresses used by DRBD. You can do this by rebooting every node, or by manually resetting the addresses by using the drbdadm CLI command on each node. Each method is described below.

This section describes configuring monitoring with Prometheus for LINSTOR Operator v2 deployments in Kubernetes. If you need to configure monitoring with Prometheus for a LINSTOR Operator v1 deployment, refer to the instructions in the Monitoring with Prometheus in Operator v1 Deployments section.

In Operator v1 deployments, the operator will set up monitoring containers along the existing components and make them available as a Service.

If you use the Prometheus Operator, the LINSTOR Operator will also set up the ServiceMonitor instances. The metrics will automatically be collected by the Prometheus instance associated to the operator, assuming watching the Piraeus namespace is enabled.

To disable exporting of metrics, set operator.satelliteSet.monitoringImage to an empty value.

If you deployed LINSTOR using Operator v2, you can upgrade LINBIT SDS by deploying the newer version of LINSTOR Operator.

How you upgrade LINBIT SDS depends on your original deployment method.

Before upgrading to a new release, you should ensure you have an up-to-date backup of the LINSTOR database. If you are using the etcd database packaged in the LINSTOR Chart, see here.

Upgrades will update to new versions of the following components: