LINSTOR was originally developed to manage DRBD resources. While you can still use LINSTOR to make managing DRBD more convenient, LINSTOR has evolved and it is often integrated with software stacks higher up to provide persistent storage more easily and more flexibly than would otherwise be possible within those stacks.

LINSTOR can be used on its own or you can integrate it with other platforms, such as Kubernetes, OpenShift, OpenNebula, OpenStack, Proxmox VE, and others. LINSTOR runs on bare-metal on-premise hardware, or you can use it within virtual machines (VMs), containers, clouds, or hybrid environments.

LINSTOR can work with the following storage provider and related technologies:

By using LINSTOR, you can work with these technologies on their own or else in various meaningful combinations.

The linstor-controller service relies on a database that holds all the configuration information for the whole cluster. A node or container running the LINSTOR controller software is responsible for resource placement, resource configuration, and orchestration of any operational processes that require a view of the whole cluster.

Multiple controllers can be used for LINSTOR, for example, when setting up a highly available LINSTOR cluster, but only one controller can be active.

As mentioned earlier, the LINSTOR controller operates on a separate plane from the data plane that it manages. You can stop the controller service, update or reboot the controller node, and still have access to your data hosted on the LINSTOR satellite nodes. While you can still access and serve the data on your LINSTOR satellite nodes, without a running controller node, you will not be able to perform any LINSTOR status or management tasks on the satellite nodes.

The linstor-satellite service runs on each node where LINSTOR consumes local storage or provides storage to services. It is stateless and receives all the information it needs from the node or container running the LINSTOR controller service. The LINSTOR satellite service runs programs like lvcreate and drbdadm. It acts like an agent on a node or in a container that carries out instructions that it receives from the LINSTOR controller node or container.

When you need to interface with LINSTOR, you can send instructions to the active LINSTOR controller by using one of its user interfaces (UIs): the LINSTOR client, or the LINBIT SDS GUI.

Both of these UIs rely on the LINSTOR REST API.

LINSTOR takes an object-oriented approach to software-defined storage (SDS). LINSTOR objects are the end result that LINSTOR presents to the user or application to consume or build upon.

The most commonly used LINSTOR objects are explained below and a full list of objects follows.

While definitions and groups are also LINSTOR objects, they are a special kind. Definition and group objects can be thought of as profiles or templates. These template objects are used to create child objects that will inherit their parent object’s attributes. They might also have attributes that can affect child objects but are not attributes of the child objects themselves. These attributes could be things such as the TCP port to use for DRBD replication or the volume number that a DRBD resource should use.

The following are some general rules for object hierarchy in LINSTOR:

This section uses diagrams to represent the hierarchical relationships between some of the most frequently used LINSTOR objects. Because of the number of LINSTOR objects and their interconnectedness, multiple diagrams are shown first rather than a single diagram, to simplify the conceptualization.

The next diagram shows the relationships between LINSTOR group objects on a single satellite node.

While the two preceding diagrams show higher-lower relationships between common LINSTOR objects, you can also think of some LINSTOR objects as having parent-child relationships. The next diagram introduces this kind of relationship between LINSTOR objects by using a storage pool definition (parent object) and a storage pool (child object) as an example. A parent object can have multiple child objects, as shown in the following diagram.

Having introduced the concept of parent-child relationships in a conceptual diagram, the next diagram is a modified version of the second diagram with some of those relationships added for groups and definitions. This modified diagram also incorporates some of the higher-lower relationships that were shown in the first diagram.

The next diagram synthesizes the relationship concepts of the preceding diagrams while also introducing new LINSTOR objects related to snapshots and connections. With the many objects and criss-crossing lines, the reason for building up to this diagram should be apparent.

Even with its seeming complexity, the preceding diagram is still a simplification and not an all-encompassing representation of the possible relationships between LINSTOR objects. As listed earlier, there are more LINSTOR objects than are shown in the diagram^[3].

The good news is that you do not need to memorize the preceding diagram to work with LINSTOR. It could be useful to refer to though if you are trying to troubleshoot attributes that you have set on LINSTOR objects and their inheritance and effects on other LINSTOR objects in your cluster.

LINSTOR is packaged in both the RPM and the DEB variants:

For further details about these packages see the Installable Components section above.

You can configure LINSTOR to encrypt storage volumes, by using LUKS (dm-crypt), as detailed in the Encrypted Volumes section of this user’s guide. Refer to the LUKS and dm-crypt projects for FIPS compliance status.

You can also configure LINSTOR to encrypt communication traffic between a LINSTOR satellite and a LINSTOR controller, by using SSL/TLS, as detailed in the Secure Satellite Connections section of this user’s guide.

LINSTOR can also interface with Self-Encrypting Drives (SEDs) and you can use LINSTOR to initialize an SED drive. LINSTOR stores the drive’s password as a property that applies to the storage pool associated with the drive. LINSTOR encrypts the SED drive password by using the LINSTOR master passphrase that you must create first.

By default, LINSTOR uses the following cryptographic algorithms:

A FIPS compliant version of LINSTOR is available for the use cases mentioned in this section. If you or your organization require FIPS compliance at this level, please contact [email protected] for details.

To use LINSTOR to create storage volumes, you will need to install a volume manager, either LVM or ZFS, if one is not already installed on your system.

If you are a LINBIT customer, you can download a LINBIT created helper script and run it on your nodes to:

Enabling LINBIT package repositories will give you access to LINBIT software packages, DRBD kernel modules, and other related software such as cluster managers and OCF scripts. You can then use a package manager to fetch, install, and manage updating installed packages.

After registering your node and enabling the drbd-9 LINBIT package repository, you can use a DEB, RPM, or YaST2 based package manager to install LINSTOR and related components.

The LINSTOR project’s GitHub page is here: https://github.com/LINBIT/linstor-server.

LINBIT also has downloadable archived files of source code for LINSTOR, DRBD, and more, available here: https://linbit.com/linbit-software-download-page-for-linstor-and-drbd-linux-driver/.

On each host contributing storage, you need to create either an LVM volume group (VG) or a ZFS zPool. The VGs and zPools identified with one LINSTOR storage pool name may have different VG or zPool names on the hosts, but do yourself a favor, for coherency, use the same VG or zPool name on all nodes.

After creating a volume group on each of your nodes, you can create a storage pool that is backed by the volume group on each of your nodes, by entering the following commands:

To list your storage pools you can enter:

or using LINSTOR abbreviated notation:

In clusters where you have only one kind of storage and the capability to hot swap storage devices, you may choose a model where you create one storage pool per physical backing device. The advantage of this model is to confine failure domains to a single storage device.

Both the Exos and LVM2 storage providers offer the option of multiple server nodes directly connected to the storage array and drives. With LVM2 the external locking service (lvmlockd) manages volume groups created with the –shared options with vgcreate. The --shared-space can be used when configuring a LINSTOR pool to use the same LVM2 volume group accessible by two or more nodes. The example below shows using the LVM2 volume group UUID as the shared space identifier for a pool accessible by nodes alpha and bravo:

Exos pools will use the Exos pool serial number by default for the shared-space identifier.

Since linstor-server 1.5.2 and a recent linstor-client, LINSTOR can create LVM/ZFS pools on a satellite for you. The linstor-client has the following commands to list possible disks and create storage pools, but such LVM/ZFS pools are not managed by LINSTOR and there is no delete command, so such action must be done manually on the nodes.

Will give you a list of available disks grouped by size and rotational(SSD/Magnetic Disk).

It will only show disks that pass the following filters:

With the create-device-pool command you can create a LVM pool on a disk and also directly add it as a storage pool in LINSTOR.

If the --storage-pool option was provided, LINSTOR will create a storage pool with the given name.

For more options and exact command usage please check the linstor-client help.

LINSTOR has the following supported storage plug-ins at the time of writing:

使用 resource create 命令，可以将资源定义显式分配给命名节点。

It is possible to have LINSTOR select nodes and storage pools to deploy a resource to, by using the linstor resource create --auto-place or linstor resource-definition auto-place commmands. This section will use the resource create --auto-place command in examples. However, you can use either command to produce the same results.

In the following example, the value after the --auto-place option tells LINSTOR how many replicas you want to have. The --storage-pool option should be obvious.

可能不太明显的是，您可以省略 --storage-pool 选项，然后LINSTOR可以自己选择一个存储池。选择遵循以下规则：

The remaining storage pools will be rated by different strategies. LINSTOR has currently three strategies:

MaxFreeSpace: This strategy maps the rating 1:1 to the remaining free space of the storage pool. However, this strategy only considers the actually allocated space (in case of thin-provisioned storage pool this might grow with time without creating new resources) MinReservedSpace: Unlike the “MaxFreeSpace”, this strategy considers the reserved space. That is the space that a thin volume can grow to before reaching its limit. The sum of reserved spaces might exceed the storage pool’s capacity, which is as overprovisioning. MinRscCount: Simply the count of resources already deployed in a given storage pool MaxThroughput: For this strategy, the storage pool’s Autoplacer/MaxThroughput property is the base of the score, or 0 if the property is not present. Every Volume deployed in the given storage pool will subtract its defined sys/fs/blkio_throttle_read and sys/fs/blkio_throttle_write property- value from the storage pool’s max throughput. The resulting score might be negative.

The scores of the strategies will be normalized, weighted and summed up, where the scores of minimizing strategies will be converted first to allow an overall maximization of the resulting score.

The weights of the strategies can be configured with the following command:

The strategy names are listed above and the weight can be an arbitrary decimal.

Finally, LINSTOR tries to find the best matching group of storage pools meeting all requirements. This step also considers other auto-placement restrictions such as --replicas-on-same, --replicas-on-different, --do-not-place-with, --do-not-place-with-regex, --layer-list, and --providers.

You can delete a resource definition by using the command:

This will remove the named resource definition from the entire LINSTOR cluster. The resource is removed from all nodes and the resource entry is marked for removal from LINSTOR’s database tables. After LINSTOR has removed the resource from all the nodes, the resource entry is removed from LINSTOR’s database tables.

You can delete a resource using the command:

Unlike deleting a resource definition, this command will only delete a LINSTOR resource from the node (or nodes) that you specify. The resource is removed from the node and the resource entry is marked for removal from LINSTOR’s database tables. After LINSTOR has removed the resource from the node, the resource entry is removed from LINSTOR’s database tables.

Deleting a LINSTOR resource may have implications for a cluster, beyond just removing the resource. For example, if the resource is backed by a DRBD layer, removing a resource from one node in a three node cluster could also remove certain quorum related DRBD options, if any existed for the resource. After removing such a resource from a node in a three node cluster, the resource would no longer have quorum as it would now only be deployed on two nodes in the three node cluster.

After running a linstor resource delete command to remove a resource from a single node, you might see informational messages such as:

Also unlike deleting a resource definition, you can delete a resource while there are existing snapshots of the resource’s storage pool. Any existing snapshots for the resource’s storage pool will persist.

You can delete a resource group by using the command:

As you might expect, this command deletes the named resource group. You can only delete a resource group if it has no associated resource definitions, otherwise LINSTOR will present an error message, such as:

To resolve this error so that you can delete the resource group, you can either delete the associated resource definitions, or your can move the resource definitions to another (existing) resource group:

You can find which resource definitions are associated with your resource group by entering the following command:

To backup the LINSTOR database to a new file named db_export in your home directory, enter the following commands:

After backing up the database, you can copy the backup file to a safe place.

The resulting database backup is a plain JSON document, containing not just the actual data, but also some metadata about when the backup was created, from which database, and other information.

Restoring the database from a previously made backup is similar to export-db from the previous section.

For example, to restore the previously made backup from the db_export file, enter the following commands:

You can only import a database from a previous backup if the currently installed version of LINSTOR is the same (or higher) version as the version that you created the backup from. If the currently installed LINSTOR version is higher than the version that the database backup was created from, when you import the backup the data will be restored with the same database scheme of the version used during the export. Then, the next time that the controller starts, the controller will detect that the database has an old scheme and it will automatically migrate the data to the scheme of the current version.

Since the exported database file contains some metadata, an exported database file can be imported into a different database type than it was exported from.

This allows the user to convert, for example, from an etcd setup to an SQL based setup. There is no special command for converting the database format. You only have to specify the correct linstor.toml configuration file by using the --config-directory argument (or updating the default /etc/linstor/linstor.toml and specifying the database type that you want to use before importing). See the LINSTOR User’s Guide for more information about specifying a database type. Regardless of the type of database that the backup was created from, it will be imported in the database type that is specified in the linstor.toml configuration file.

For configuring the highly available (HA) storage, you can use LINSTOR itself. One of the benefits of having the storage under LINSTOR control is that you can easily extend the HA storage to new cluster nodes.

First, create a new resource that is 200MiB in size and set the necessary DRBD options, as shown in the example commands below. You will need to adapt the storage pool name to match an existing storage pool in your environment.

It is crucial that your cluster qualifies for auto-quorum and uses the io-error policy (see Section [s-linstor-auto-quorum]), and that auto-promote is disabled.

From now on it is assumed that the resource’s name is linstor_db. After creating the linstor_db resource, you can move the LINSTOR database to the new storage and create a systemd mount service. First, stop the current controller service and disable it, as it will be managed by DRBD Reactor later.

Next, create the systemd mount service.

Copy the /etc/systemd/system/var-lib-linstor.mount mount file to all the cluster nodes that you want to have the potential to run the LINSTOR controller service (standby controller nodes). Again, do not systemctl enable any of these services because DRBD Reactor will manage them.

The next step is to install LINSTOR controllers on all nodes that have access to the linstor_db DRBD resource (as they need to mount the DRBD volume) and which you want to become a possible LINSTOR controller. It is important that the controllers are manged by drbd-reactor, so verify that the linstor-controller.service is disabled on all nodes! To be sure, execute systemctl disable linstor-controller on all cluster nodes and systemctl stop linstor-controller on all nodes except the one it is currently running from the previous step. Also verify that you have set chattr +i /var/lib/linstor on all potential controller nodes if you use LINSTOR version equal or greater to 1.14.0.

For starting and stopping the mount service and the linstor-controller service we use drbd-reactor. Install this component on all nodes that could become a LINSTOR controller and edit their /etc/drbd-reactor.d/linstor_db.toml configuration file. It should contain an enabled promoter plug-in section like this:

Depending on your requirements you might also want to set an on-stop-failure action and set stop-services-on-exit.

After that restart drbd-reactor and enable it on all the nodes you configured it.

Check that there are no warnings from drbd-reactor service in the logs by running systemctl status drbd-reactor. As there is already an active LINSTOR controller things will just stay the way they are. Run drbd-reactorctl status linstor_db to check the health of the linstor_db target unit.

The last but nevertheless important step is to configure the LINSTOR satellite services to not delete (and then regenerate) the resource file for the LINSTOR controller DB at its startup. Do not edit the service files directly, but use systemctl edit. Edit the service file on all nodes that could become a LINSTOR controller and that are also LINSTOR satellites.

After this change you should execute systemctl restart linstor-satellite on all satellite nodes.

NVMe-oF/NVMe-TCP允许LINSTOR将无盘资源连接到具有相同资源的节点，数据存储在NVMe结构上。这就带来了这样一个优势：通过网络访问数据，无需使用本地存储就可以装载资源。在这种情况下，LINSTOR不使用DRBD，因此不会复制LINSTOR提供的NVMe资源，数据存储在一个节点上。

To use NVMe-oF/NVMe-TCP with LINSTOR the package nvme-cli needs to be installed on every node which acts as a satellite and will use NVMe-oF/NVMe-TCP for a resource. For example, on a DEB-based system, to install the package, enter the following command:

要使资源使用NVMe-oF/NVMe-TCP，必须在创建资源定义时提供一个附加参数：

To use NVMe-TCP rather than the default NVMe-oF, the following property needs to be set:

The property NVMe/TRType can alternatively be set on resource-group or controller level.

Next, create the volume-definition for our resource:

在节点上创建资源之前，必须知道数据将在本地存储在哪里，以及哪个节点通过网络访问它。

首先，我们在存储数据的节点上创建资源：

在将通过网络访问资源数据的节点上，必须将资源定义为无盘：

现在，您可以在一个节点上装载资源 nvmedata。

Since version 1.5.0 the additional Layer openflex can be used in LINSTOR. From LINSTOR’s perspective, the OpenFlex Composable Infrastructure takes the role of a combined layer acting as a storage layer (like LVM) and also providing the allocated space as an NVMe target. OpenFlex has a REST API which is also used by LINSTOR to operate with.

As OpenFlex combines concepts of LINSTOR’s storage as well as NVMe-layer, LINSTOR was added both, a new storage driver for the storage pools as well as a dedicated openflex layer which uses the mentioned REST API.

In order for LINSTOR to communicate with the OpenFlex-API, LINSTOR needs some additional properties, which can be set once on controller level to take LINSTOR-cluster wide effect:

Once that is configured, we can now create LINSTOR objects to represent the OpenFlex architecture. The theoretical mapping of LINSTOR objects to OpenFlex objects are as follows: Obviously an OpenFlex storage pool is represented by a LINSTOR storage pool. As the next thing above a LINSTOR storage pool is already the node, a LINSTOR node represents an OpenFlex storage device. The OpenFlex objects above storage device are not mapped by LINSTOR.

When using NVMe, LINSTOR was designed to run on both sides, the NVMe target as well as on the NVMe initiator side. In the case of OpenFlex, LINSTOR cannot (or even should not) run on the NVMe target side as that is completely managed by OpenFlex. As LINSTOR still needs nodes and storage pools to represent the OpenFlex counterparts, the LINSTOR client was extended with special node create commands since 1.0.14. These commands not only accept additionally needed configuration data, but also starts a “special satellite” besides the already running controller instance. These special satellites are completely LINSTOR managed. They will shut down when the controller shuts down and will be started again when the controller starts.

The new client command for creating a “special satellite” representing an OpenFlex storage device is:

The arguments are as follows:

The last step of the configuration is the creation of LINSTOR storage pools:

Once all necessary storage pools are created in LINSTOR, the next steps are similar to the usage of using an NVMe resource with LINSTOR. Here is a complete example:

A DM-Writecache device is composed of two devices: one storage device and one cache device. LINSTOR can setup such a writecache device, but needs some additional information, like the storage pool and the size of the cache device.

The two properties set in the examples are mandatory, but can also be set on controller level which would act as a default for all resources with WRITECACHE in their --layer-list. However, please note that the Writecache/PoolName refers to the corresponding node. If the node does not have a storage pool named pmempool you will get an error message.

The 4 mandatory parameters required by DM-Writecache are either configured through a property or figured out by LINSTOR. The optional properties listed in the mentioned link can also be set through a property. Please see linstor controller set-property --help for a list of Writecache/* property-keys.

使用 --layer-list DRBD,WRITECACHE,STORAGE 将DRBD配置为使用外部元数据时，只有备份设备将使用writecache，而不是保存外部元数据的设备。

LINSTOR can also setup a DM-Cache device, which is very similar to the DM-Writecache from the previous section. The major difference is that a cache device is composed by three devices: one storage device, one cache device and one meta device. The LINSTOR properties are quite similar to those of the writecache but are located in the Cache namespace:

Please see linstor controller set-property --help for a list of Cache/* property-keys and default values for omitted properties.

Using --layer-list DRBD,CACHE,STORAGE while having DRBD configured to use external metadata, only the backing device will use a cache, not the device holding the external metadata.

The storage layer will provide new devices from well known volume managers like LVM, ZFS or others. Every layer combination needs to be based on a storage layer, even if the resource should be diskless – for that type there is a dedicated diskless provider type.

For a list of providers with their properties please see Storage Providers.

For some storage providers LINSTOR has special properties:

A storage pool with the type remote SPDK can only be created on a “special satellite”. For this you first need to start a new satellite using the command:

This will start a new satellite instance running on the same machine as the controller. This special satellite will do all the REST based RPC communication towards the remote SPDK proxy. As the help message of the LINSTOR command shows, the administrator might want to use additional settings when creating this special satellite:

The difference between the --api-* and their corresponding --api-\*-env versions is that the version with the -env ending will look for an environment variable containing the actual value to use whereas the --api-\* version directly take the value which is stored in the LINSTOR property. Administrators might not want to save the --api-pw in plain text, which would be clearly visible using commands like linstor node list-property <nodeName>.

Once that special satellite is up and running the actual storage pool can be created:

Whereas node_name is self-explanatory, name is the name of the LINSTOR storage pool and driver_pool_name refers to the SPDK logical volume store.

Once this remotespdk storage pool is created the remaining procedure is quite similar as using NVMe: First the target has to be created by creating a simple “diskful” resource followed by a second resource having the --nvme-initiator option enabled.

To use multiple network paths for DRBD setups, the PrefNic property is not sufficient. Instead the linstor node interface and linstor resource-connection path commands should be used, as shown below.

The first four commands in the example define a network interface (nic1 and nic2) for each node (alpha and bravo) by specifying the network interface’s IP address. The last two commands create network path entries in the DRBD .res file that LINSTOR generates. This is the relevant part of the resulting .res file:

下面是有关命令的详细信息。

在LINSTOR可以加密任何卷之前，需要创建主密码。这可以通过linstor客户端完成。

`crypt create-passphrase`将等待用户输入初始主密码（因为所有其他crypt命令都没有参数）。

如果您想更改主密码，可以使用以下方法：

luks 层可以在创建资源定义或资源本身时添加，而建议使用前一种方法，因为它将自动应用于从该资源定义创建的所有资源。

要输入主密码（在controller重新启动后），请使用以下命令：

It is possible to automate the process of creating and re-entering the master passphrase.

To use this, either an environment variable called MASTER_PASSPHRASE or an entry in /etc/linstor/linstor.toml containing the master passphrase has to be created.

The required linstor.toml looks like this:

If either one of these is set, then every time the controller starts it will check whether a master passphrase already exists. If there is none, it will create a new master passphrase as specified. Otherwise, the controller enters the passphrase.

Some evacuation cases may need special planning. For example, if you are evacuating more than one node, you can exclude the nodes from participating in LINSTOR’s resource autoplacer. You can do this by using the following command on each node that you want to evacuate:

This ensures that LINSTOR will not place resources from a node that you are evacuating onto another node that you plan on evacuating.

If you already ran a node evacuate command that has either completed or still has resources in an “Evacuating” state, you can remove the “Evacuating” state from a node by using the node restore command. This will work so long as you have not yet run a node delete command.

After restoring the node, you should use the node set-property <node_name> AutoplaceTarget true command, if you previously set the AutoplaceTarget property to “false”. This way, LINSTOR can again place resources onto the node automatically, to fulfill placement count properties that you might have set for resources in your cluster.

假设在某些节点上放置了名为 resource1 的资源定义，则可以按如下方式创建快照：

这将在资源所在的所有节点上创建快照。LINSTOR将确保即使在资源处于活动使用状态时也创建一致的快照。

Setting the resource-definition property AutoSnapshot/RunEvery LINSTOR will automatically create snapshots every X minute. The optional property AutoSnapshot/Keep can be used to clean-up old snapshots which were created automatically. No manually created snapshot will be cleaned-up / deleted. If AutoSnapshot/Keep is omitted (or ⇐ 0), LINSTOR will keep the last 10 snapshots by default.

以下步骤将快照还原到新资源。即使原始资源已从创建快照的节点中删除，也可能发生这种情况。

首先使用与快照中的卷匹配的卷定义新资源：

此时，如果需要，可以应用其他配置。然后，准备好后，根据快照创建资源：

这将在快照所在的所有节点上放置新资源。也可以显式选择放置资源的节点；请参阅帮助（ linstor snapshot resource restore -h ).

LINSTOR可以将资源回滚到快照状态。回滚时资源不能被使用。也就是说，它不能安装在任何节点上。如果资源正在使用中，请考虑是否可以通过restoring the snapshot来实现您的目标。

回滚执行如下：

资源只能回滚到最近的快照。要回滚到旧快照，请首先删除中间快照。

可以按如下方式删除现有快照：

Snapshots can be shipped between LINSTOR nodes or between different LINSTOR clusters, as well as to an S3 storage such as Amazon S3 or min.io.

The following tools need to be installed on the satellites that are going to send or receive snapshots:

You create a backup shipping schedule by using the LINSTOR client schedule create command and defining the frequency of backup shipping using cron syntax. You also need to set options that name the schedule and define various aspects of the backup shipping, such as on-failure actions, the number of local and remote backup copies to keep, and whether to also schedule incremental backup shipping.

At a minimum, the command needs a schedule name and a full backup cron schema to create a backup shipping schedule. An example command would look like this:

You can modify a backup shipping schedule by using the LINSTOR client schedule modify command. The syntax for the command is the same as that for the schedule create command. The name that you specify with the schedule modify command must be an already existing backup schedule. Any options to the command that you do not specify will retain their existing values. If you want to set the keep-local or keep-remote options back to their default values, you can set them to “all”. If you want to set the max-retries option to its default value, you can set it to “forever”.

Your physical storage is not infinite and your remote storage has a cost, so you will likely want to set limits on the number of snapshots and backups you keep.

Both the --keep-remote and --keep-local options deserve special mention as they have implications beyond what may be obvious. Using these options, you specify how many snapshots or full backups should be kept, either on the local source or the remote destination.

You can list your backup shipping schedules by using the linstor schedule list command.

For example:

The LINSTOR client schedule delete command completely deletes a backup shipping schedule LINSTOR object. The command’s only argument is the schedule name that you want to delete. If the deleted schedule is currently creating or shipping a backup, the scheduled shipping process is stopped. Depending on at which point the process stops, a snapshot, or a backup, or both, might not be created and shipped.

This command does not affect previously created snapshots or successfully shipped backups. These will be retained until they are manually deleted.

You can use the LINSTOR client backup schedule enable command to enable a previously created backup shipping schedule. The command has the following syntax:

To disable a previously enabled backup shipping schedule, you use the LINSTOR client backup schedule disable command. The command has the following syntax:

If you include the option specifying either a resource group or resource definition, as described in the backup schedule enable command example above, then you disable the schedule only for that resource group or resource definition.

For example, if you omitted specifying a resource group or resource definition in an earlier backup schedule enable command, LINSTOR would schedule backup shipping for all its deployed resources that can make snapshots. Your disable command would then only affect the resource group or resource definition that you specify with the command. The backup shipping schedule would still apply to any deployed LINSTOR resources besides the specified resource group or resource definition.

The same as for the backup schedule enable command, if you specify neither a resource group nor a resource definition, then LINSTOR disables the backup shipping schedule at the controller level for all deployed LINSTOR resources.

You can use the linstor backup schedule delete command to granularly delete either a specified resource definition or a resource group from a backup shipping schedule, without deleting the schedule itself. This command has the same syntax and arguments as the backup schedule disable command. If you specify neither a resource group nor a resource definition, the backup shipping schedule you specify will be deleted at the controller level.

It may be helpful to think about the backup schedule delete command as a way that you can remove a backup shipping schedule-remote pair from a specified LINSTOR object level, either a resource definition, a resource group, or at the controller level if neither is specified.

The backup schedule delete command does not affect previously created snapshots or successfully shipped backups. These will be retained until they are manually deleted, or until they are removed by the effects of a still applicable keep-local or keep-remote option.

You might want to use this command when you have disabled a backup schedule for multiple LINSTOR object levels and later want to affect a granular change, where a backup schedule enable command might have unintended consequences.

For example, consider a scenario where you have a backup schedule-remote pair that you enabled at a controller level. This controller has a resource group, myresgroup that has several resource definitions, resdef1 through resdef9, under it. For maintenance reasons perhaps, you disable the schedule for two resource definitions, resdef1 and resdef2. You then realize that further maintenance requires that you disable the backup shipping schedule at the resource group level, for your myresgroup resource group.

After completing some maintenance, you are able to enable the backup shipping schedule for resdef3 through resdef9, but you are not yet ready to resume (enable) backup shipping for resdef1 and resdef2. You can enable backup shipping for each resource definition individually, resdef3 through resdef9, or you can use the backup schedule delete command to delete the backup shipping schedule from the resource group, myresgroup. If you use the backup schedule delete command, backups of resdef3 through resdef9 will ship again because the backup shipping schedule is enabled at the controller level, but resdef1 and resdef2 will not ship because the backup shipping schedule is still disabled for them at the resource definition level.

When you complete your maintenance and are again ready to ship backups for resdef1 and resdef2, you can delete the backup shipping schedule for those two resource definitions to return to your starting state: backup shipping scheduled for all LINSTOR deployed resources at the controller level. To visualize this it may be helpful to refer to the decision tree diagram for how LINSTOR decides whether or not to ship a backup in the How the LINSTOR Controller Determines Scheduled Backup Shipping subsection.

You can list backup schedules by resource, using the LINSTOR client schedule list-by-resource command. This command will show LINSTOR resources and how any backup shipping schedules apply and to which remotes they are being shipped. If resources are not being shipped then the command will show:

If resources have schedule-remote-pairs and are being shipped, the command output will show when the last backup was shipped and when the next backup is scheduled to ship. It will also show whether the next and last backup shipments were full or incremental backups. Finally, the command will show when the next planned incremental (if any) and full backup shipping will occur.

You can use the --active-only flag with the schedule list-by-resource command to filter out all resources that are not being shipped.

To determine if the LINSTOR Controller will ship a deployed LINSTOR resource with a certain backup schedule for a given remote destination, the LINSTOR Controller uses the following logic:

As the diagram shows, enabled or disabled backup shipping schedules have effect in the following order:

A backup shipping schedule-remote pair that is enabled or disabled at a preceding level will override the enabled or disabled status for the same schedule-remote pair at a later level.

To determine how a LINSTOR resource will be affected by scheduled backup shipping, you can use the LINSTOR client schedule list-by-resource-details command for a specified LINSTOR resource.

The command will output a table that shows on what LINSTOR object level a backup shipping schedule is either not set (empty cell), enabled, or disabled.

By using this command, you can determine on which level you need to make a change to enable, disable, or delete scheduled backup shipping for a resource.

Example output could look like this:

The LINSTOR resource object is an exception to the general syntax for setting DRBD options for LINSTOR objects. With the LINSTOR resource object, you can use the drbd-peer-options to set DRBD options at the connection level between the two nodes that you specify. Specifying drbd-peer-options for a LINSTOR resource object between two nodes is equivalent to using the`linstor resource-connection drbd-peer-options` for a resource between two nodes.

For example, to set the DRBD maximum buffer size to 8192 at a connection level, for a resource named backups, between two nodes, node-0 and node-1, enter:

The command above is equivalent to the following:

Indeed, when using the linstor --curl command to examine the two commands actions on the LINSTOR REST API, the output is identical:

The connection section of the LINSTOR-generated resource file backups.res on node-0 will look something like this:

You can use the drbd-peer-options argument to set DRBD options at a connection level, between two nodes, for example:

The preceding command would set the DRBD ping-timeout option to 29.9 seconds at a connection level between two nodes, node-0 and node-1.

You can verify a LINSTOR object’s set properties by using the list-properties command, for example:

To remove a previously set DRBD option, prefix the option name with unset-. For example:

The same syntax applies to any drbd-peer-options set either on a LINSTOR resource, resource connection, or node connection. For example:

Removing a DRBD option or DRBD peer option will return the option to its default value. Refer to the linstor <LINSTOR_object> drbd-options --help (or drbd-peer-options --help) command output for the default values of options. You can also refer to the drbd.conf-9.0 man page to get information about DRBD options. === Adding and Removing Disks

LINSTOR can convert resources between diskless and having a disk. This is achieved with the resource toggle-disk command, which has syntax similar to resource create.

For instance, add a disk to the diskless resource backups on ‘alpha’:

Remove this disk again:

To move a resource between nodes without reducing redundancy at any point, LINSTOR’s disk migrate feature can be used. First create a diskless resource on the target node, and then add a disk using the --migrate-from option. This will wait until the data has been synced to the new disk and then remove the source disk.

For example, to migrate a resource backups from ‘alpha’ to ‘bravo’:

LINSTOR can also be configured to automatically enable the above mentioned Proxy connection between two nodes. For this automation, LINSTOR first needs to know on which site each node is.

As the Site property might also be used for other site-based decisions in future features, the DrbdProxy/AutoEnable also has to be set to true:

This property can also be set on node, resource-definition, resource and resource-connection level (from left to right in increasing priority, whereas the controller is the left-most, that is, the least prioritized level). Once this initialization steps are completed, every newly created resource will automatically check if it has to enable DRBD proxy to any of its peer-resources. === External Database Providers

It is possible to have LINSTOR working with an external database provider like PostgreSQL, MariaDB and since version 1.1.0 even etcd key value store is supported. To use an external database there are a few additional steps to configure. You have to create a DB/Schema and user to use for linstor, and configure this in the /etc/linstor/linstor.toml. ==== PostgreSQL

A sample PostgreSQL linstor.toml looks like this:

A sample MariaDB linstor.toml looks like this:

etcd is a distributed key-value store that makes it easy to keep your LINSTOR database distributed in a HA-setup. The etcd driver is already included in the LINSTOR-controller package and only needs to be configured in the linstor.toml.

More information about how to install and configure etcd can be found here: etcd docs

And here is a sample [db] section from the linstor.toml:

The LINSTOR-Controller also provides a /health HTTP path that will simply return HTTP-Status 200 if the controller can access its database and all services are up and running. Otherwise it will return HTTP error status code 500 Internal Server Error. === Secure Satellite Connections

It is possible to have the LINSTOR use SSL/TLS secure TCP connection between controller and satellite connections. Without going into further details on how Java’s SSL/TLS engine works we will give you command line snippets using the keytool from Java’s runtime environment on how to configure a three node setup using secure connections.

The node setup looks like this:

Node alpha is the just the controller. Node bravo and node charlie are just satellites.

Here are the commands to generate such a keystore setup, values should of course be edited for your environment.

Now just start controller and satellites and add the nodes with --communication-type SSL. === Configuring LDAP Authentication

You can configure LINSTOR to use LDAP authentication to limit access to the LINSTOR Controller. This feature is disabled by default but you can enable and configure it by editing the LINSTOR configuration TOML file. After editing the configuration file, you will need to restart the linstor-controller.service. An example LDAP section within the configuration file looks like this:

After configuring the LINSTOR Controller to authenticate users through LDAP (or LDAPS), and the LINSTOR REST API HTTPS, you will need to enter LINSTOR commands as follows:

If you have configured LDAP authentication without also configuring LINSTOR REST API HTTPS, you will need to explicitly enable password authentication over HTTP, by using the --allow-insecure-path flag with your linstor commands. This is not recommended outside of a secured and isolated LAN, as you will be sending credentials in plain text.

The LINSTOR Controller will prompt you for the user’s password, in each of the above examples. You may optionally use the --password argument to supply the user’s password on the command line, with all the warnings of caution that would go along with doing so. === Automatisms for DRBD Resources

This section details some of LINSTOR’s automatisms for DRBD resources. ==== Auto-Quorum Policies

LINSTOR automatically configures quorum policies on resources when quorum is achievable. This means, whenever you have at least two diskful and one or more diskless resource assignments, or three or more diskful resource assignments, LINSTOR will enable quorum policies for your resources automatically.

Inversely, LINSTOR will automatically disable quorum policies whenever there are less than the minimum required resource assignments to achieve quorum.

This is controlled through the, DrbdOptions/auto-quorum, property which can be applied to the linstor-controller, resource-group, and resource-definition. Accepted values for the DrbdOptions/auto-quorum property are disabled, suspend-io, and io-error.

Setting the DrbdOptions/auto-quorum property to disabled will allow you to manually, or more granularly, control the quorum policies of your resources should you want to.

For example, to manually set the quorum policies of a resource-group named my_ssd_group, you would use the following commands:

You may want to disable DRBD’s quorum features completely. To do that, you would need to first disable DrbdOptions/auto-quorum on the appropriate LINSTOR object, and then set the DRBD quorum features accordingly. For example, use the following commands to disable quorum entirely on the my_ssd_group resource-group:

If a satellite is offline for a prolonged period of time, LINSTOR can be configured to declare that node as evicted. This triggers an automated reassignment of the affected DRBD resources to other nodes to ensure a minimum replica count is kept.

This feature uses the following properties to adapt the behaviour.

After the linstor-controller loses the connection to a satellite, aside from trying to reconnect, it starts a timer for that satellite. As soon as that timer exceeds DrbdOptions/AutoEvictAfterTime and all of the DRBD-connections to the DRBD-resources on that satellite are broken, the controller will check whether or not DrbdOptions/AutoEvictMaxDisconnectedNodes has been met. If it hasn’t, and DrbdOptions/AutoEvictAllowEviction is true for the node in question, the satellite will be marked as EVICTED. At the same time, the controller will check for every DRBD-resource whether the number of resources is still above DrbdOptions/AutoEvictMinReplicaCount. If it is, the resource in question will be marked as DELETED. If it isn’t, an auto-place with the settings from the corresponding resource-group will be started. Should the auto-place fail, the controller will try again later when changes that might allow a different result, such as adding a new node, have happened. Resources where an auto-place is necessary will only be marked as DELETED if the corresponding auto-place was successful.

The evicted satellite itself will not be able to reestablish connection with the controller. Even if the node is up and running, a manual reconnect will fail. It is also not possible to delete the satellite, even if it is working as it should be. The satellite can, however, be restored. This will remove the EVICTED-flag from the satellite and allow you to use it again. Previously configured network interfaces, storage pools, properties and similar entities as well as non-DRBD-related resources and resources that could not be autoplaced somewhere else will still be on the satellite. To restore a satellite, use

Should you want to instead throw everything that once was on that node, including the node itself, away, you need to use the node lost command instead. ==== Auto-Diskful and Related Options

You can set the LINSTOR auto-diskful and auto-diskful-allow-cleanup properties for various LINSTOR objects, for example, a resource definition, to have LINSTOR help automatically make a Diskless node Diskful and perform appropriate cleanup actions afterwards. This is useful when a Diskless node has been in a Primary state for a DRBD resource for more than a specified number of minutes. This could happen in cases where you integrate LINSTOR managed storage with other orchestrating and scheduling platforms, such as OpenStack, OpenNebula, and others. On some platforms that you integrate LINSTOR with, you might not have a way to influence where in your cluster a storage volume will be used. The auto-diskful options give you a way to use LINSTOR to sensibly delegate the roles of your storage nodes in response to an integrated platform’s actions that are beyond your control. ===== Setting the Auto-Diskful Option

By setting the DrbdOptions/auto-diskful option on a LINSTOR resource definition , you are configuring the LINSTOR controller to make a Diskless DRBD resource Diskful if the resource has been in a DRBD Primary state for more than the specified number of minutes. After the specified number of minutes, LINSTOR will automatically use the resource toggle-disk command to toggle the resource state on the Diskless node, for the given resource.

To set this property, for example, on a LINSTOR resource definition named myres with a threshold of five minutes, enter the command:

These LINSTOR properties may be set using the set-property command and may be set on the following objects: volume, storage pool, resource, controller, or node. You can also set these QoS properties on resource groups, volume groups, resource definitions, or volume definitions. When you set a QoS property on a group or definition, resources created from the group or definition will inherit the QoS settings.

The following example shows creating a resource group, then creating a volume group, then applying QoS settings to the volume group, and then spawning resources from the resource group. A verification command will show that the spawned resources inherit the QoS settings. The example uses an assumed previously created LINSTOR storage pool named pool1. You will need to replace this name with a storage pool name that exists in your environment.

To verify that the spawned resources inherited the QoS setting, you can show the contents of the corresponding sysfs file, on a node that contributes storage to the storage pool.

A single LINSTOR volume can be composed of multiple DRBD devices. For example, DRBD with external metadata will have three backing devices: a data (storage) device, a metadata device, and the composite DRBD device (volume) provided to LINSTOR. If the data and metadata devices correspond to different backing disks, then if you set a sysfs property for such a LINSTOR volume, only the local data (storage) backing device will receive the property value in the corresponding /sys/fs/cgroup/blkio/ file. Neither the device backing DRBD’s metadata, nor the composite backing device provided to LINSTOR would receive the value. However, when DRBD’s data and its metadata share the same backing disk, QoS settings will affect the performance of both data and metadata operations.

In case a LINSTOR resource definition has an nvme-target as well as an nvme-initiator resource, both data (storage) backing devices of each node will receive the sysfs property value. In case of the target, the data backing device will be the volume of either LVM or ZFS, whereas in case of the initiator, the data backing device will be the connected nvme-device, regardless of which other LINSTOR layers, such as LUKS, NVMe, DRBD, and others (see Using LINSTOR Without DRBD), are above that.

在命令行中列出可用命令的一种快速方法是键入 linstor 。

Further information about subcommands (e.g., list-nodes) can be retrieved in two ways:

当LINSTOR以交互模式（LINSTOR interactive）执行时，使用 help 子命令尤其有用。

LINSTOR最有用的特性之一是它丰富的tab-completion，它基本上可以用来完成LINSTOR所知道的每个对象（例如，节点名、IP地址、资源名等等）。在下面的示例中，我们将展示一些可能的完成及其结果：

如果制表符完成不正常，请尝试获取相应的文件：

For zsh shell users, the linstor-client command can generate a zsh compilation file, that has basic support for command and argument completion.

If something goes wrong and you need help finding the cause of the issue, you can use

The command above will create a new sos-report in /var/log/linstor/controller/ on the controller node. Alternatively you can use

which will create a new sos-report and additionally downloads that report to the local machine into your current working directory.

This sos-report contains logs and useful debug-information from several sources (Linstor-logs, dmesg, versions of external tools used by LINSTOR, ip a, database dump and many more). These information are stored for each node in plain text in the resulting .tar.gz file.

如需社区帮助，请订阅我们的邮件列表： https://lists.linbit.com/listinfo/drbd-user

要提交bug或功能请求，请查看我们的GitHub页面 https://GitHub.com/linbit

Alternatively, if you need to purchase remote installation services, 24/7 support, access to certified repositories, or feature development, please contact us: +1-877-454-6248 (1-877-4LINBIT) , International: +43-1-8178292-0 | [email protected]

To deploy the Operator, create a kustomization.yaml file. This will declare your pull secret for drbd.io and allow you to pull in the Operator deployment. The Operator will be deployed in a new namespace linbit-sds. Make sure to replace MY_LINBIT_USER and MY_LINBIT_PASSWORD with your own credentials. You can find the latest releases on charts.linstor.io.

Then, apply the kustomization.yaml file, by using kubectl command, and wait for the Operator to start:

The Operator is now ready to deploy LINBIT SDS for Kubernetes.

Deploying LINBIT SDS for Kubernetes with the Operator is as simple as creating a new LinstorCluster resource and waiting for the Operator to complete the setup:

By default, LINBIT SDS for Kubernetes does not configure any storage. To add storage, you can configure a LinstorSatelliteConfiguration, which the Operator uses to configure one or more satellites.

The following example creates a simple FILE_THIN pool and it does not require any additional set up on the host:

Other types of storage pools can be configured as well. Refer to the examples upstream.

By configuring key and certificate based encryption, you can make communication between certain LINSTOR components, for example, between LINSTOR satellite nodes and a LINSTOR controller node, or between the LINSTOR client and the LINSTOR API, more secure.

After deploying LINBIT SDS for Kubernetes, you can continue with the [s-kubernetes-basic-configuration-and-deployment], Configuring the DRBD Module Loader in Operator v2 Deployments, Using the Host Network for DRBD Replication in Operator v2 Deployments sections in this chapter, or refer to the available tutorials in the upstream Piraeus project. === Deploying LINSTOR Operator v1

The Operator v1 is installed using a Helm v3 chart as follows:

The LINSTOR controller can use the Kubernetes API directly to persist its cluster state. To enable this back end, use the following override file during the chart installation:

You can use the pv-hostpath Helm templates to create hostPath persistent volumes. Create as many PVs as needed to satisfy your configured etcd replicas (default 1).

Create the hostPath persistent volumes, substituting cluster node names accordingly in the nodes= option:

By default, a PV is created on every control-plane node. You can manually select the storage nodes by passing --set "nodes={<NODE0>,<NODE1>,<NODE2>}" to the install command. NOTE: The correct value to reference the node is the value of the kubernetes.io/hostname label. You can list the value for all nodes by running kubectl get nodes -o custom-columns="Name:{.metadata.name},NodeName:{.metadata.labels['kubernetes\.io/hostname']}" ==== Using an Existing Database

LINSTOR can connect to an existing PostgreSQL, MariaDB or etcd database. For instance, for a PostgreSQL instance with the following configuration:

The Helm chart can be configured to use this database rather than deploying an etcd cluster, by adding the following to the Helm install command:

The LINSTOR Operator v1 can automate some basic storage set up for LINSTOR.

The default communication between LINSTOR components is not secured by TLS. If this is needed for your setup, choose one of three methods: // “cert-manager” is a product name so keep the original case

All the below examples use the following sp-values.yaml file. Feel free to adjust this for your uses and environment. See [Configuring storage pool creation] for further details.

Install with LINSTOR storage-pools defined at install through sp-values.yaml, persistent hostPath volumes, three etcd replicas, and by compiling the DRBD kernel modules for the host kernels.

This should be adequate for most basic deployments. Note that this deployment is not using the pre-compiled DRBD kernel modules just to make this command more portable. Using the pre-compiled binaries will make for a much faster install and deployment. Using the Compile option would not be suggested for use in a large Kubernetes clusters.

Install with LINSTOR storage-pools defined at install through sp-values.yaml, use an already created PostgreSQL DB (preferably clustered), rather than etcd, and use already compiled kernel modules for DRBD.

The PostgreSQL database in this particular example is reachable through a service endpoint named postgres. PostgreSQL itself is configured with POSTGRES_DB=postgresdb, POSTGRES_USER=postgresadmin, and POSTGRES_PASSWORD=admin123

To protect the storage infrastructure of the cluster from accidentally deleting vital components, it is necessary to perform some manual steps before deleting a Helm deployment.

The Helm charts provide a set of further customization options for advanced use cases.

To create a high-availability deployment of all components within a LINSTOR Operator v1 deployment, consult the upstream guide The default values are chosen so that scaling the components to multiple replicas ensures that the replicas are placed on different nodes. This ensures that a single node failures will not interrupt the service. NOTE: If you have deployed LINBIT SDS in Kubernetes by using the LINSTOR Operator v2, high availability is built into the deployment by default. ===== Fast Workload Failover Using the High Availability Controller

When node failures occur, Kubernetes is very conservative in rescheduling stateful workloads. This means it can take more than 15 minutes for Pods to be moved from unreachable nodes. With the information available to DRBD and LINSTOR, this process can be sped up significantly.

The LINSTOR High Availability Controller (HA Controller) speeds up the failover process for stateful workloads using LINSTOR for storage. It monitors and manages any Pod that is attached to at least one DRBD resource.

For the HA Controller to work properly, you need quorum, that is at least three replicas (or two replicas + one diskless tiebreaker). If using lower replica counts, attached Pods will be ignored and are not eligible for faster failover.

The HA Controller is packaged as a Helm chart, and can be deployed using:

If you are using the HA Controller in your cluster you can set additional parameters in all StorageClasses. These parameters ensure that the volume is not accidentally remounted as read-only, leading to degraded Pods.

To exempt a Pod from management by the HA Controller, add the following annotation to the Pod:

To create a backup of the etcd database (in LINSTOR Operator v1 deployments) and store it on your control host, enter the following commands:

These commands will create a file save.db on the machine you are running kubectl from. === Deploying with an External LINSTOR Controller

The Operator can configure the satellites and CSI plugin to use an existing LINSTOR setup. This can be useful in cases where the storage infrastructure is separate from the Kubernetes cluster. Volumes can be provisioned in diskless mode on the Kubernetes nodes while the storage nodes will provide the backing disk storage. ==== Operator v2 Deployment with an External LINSTOR Controller ===== Configuring the LinstorCluster Resource

To use an externally managed LINSTOR cluster, specify the URL of the LINSTOR controller in the LinstorCluster resource in a YAML configuration and apply it to your deployment. In the following example, the LINSTOR controller is reachable at http://linstor-controller.example.com:3370.

Normally the pod network is not reachable from outside the Kubernetes cluster. In this case the external LINSTOR controller would not be able to communicate with the satellites in the Kubernetes cluster. For this reason, you need to configure your satellites to use host networking.

To use host networking, deploy a LinstorSatelliteConfiguration resource by applying the following YAML configuration to your deployment:

To skip the creation of a LINSTOR controller deployment and configure the other components to use your existing LINSTOR controller, use the following options when running helm install:

After all pods are ready, you should see the Kubernetes cluster nodes as satellites in your LINSTOR setup.

Create a storage class referencing an existing storage pool on your storage nodes.

You can provision new volumes by creating PVCs using your storage class. The volumes will first be placed only on nodes with the given storage pool, that is, your storage infrastructure. Once you want to use the volume in a pod, LINSTOR CSI will create a diskless resource on the Kubernetes node and attach over the network to the diskful resource. === 在Kubernetes中与LINSTOR互动

The controller pod includes a LINSTOR Client, making it easy to interact directly with LINSTOR. For instance:

To simplify entering LINSTOR client commands within a Kubernetes deployment, you can use the kubectl-linstor utility. This utility is available from the upstream Piraeus datastore project. To download it, enter the following commands on your Kubernetes control plane node:

To install the utility, first extract it and then move the extracted executable file to a directory in your $PATH, for example, /usr/bin. Then you can use kubectl-linstor to get access to the complete LINSTOR CLI.

It also expands references to PVCs to the matching LINSTOR resource.

It also expands references of the form pod:[<namespace>/]<podname> into a list resources in use by the pod. This should only be necessary for investigating problems and accessing advanced functionality. Regular operation such as creating volumes should be achieved through the Kubernetes integration. === Getting Started with LINBIT SDS Storage in Kubernetes

Once all linstor-csi Pods are up and running, you can provision volumes using the usual Kubernetes workflows.

Configuring the behavior and properties of LINSTOR volumes deployed through Kubernetes is accomplished using Kubernetes StorageClass objects.

Here below is the simplest practical StorageClass that can be used to deploy volumes:

You can create the storage class with the following command:

Now that your storage class is created, you can now create a persistent volume claim (PVC) which can be used to provision volumes known both to Kubernetes and LINSTOR:

You can create the PersistentVolumeClaim with the following command:

This will create a PersistentVolumeClaim, but no volume will be created just yet. The storage class we used specified volumeBindingMode: WaitForFirstConsumer, which means that the volume is only created once a workload starts using it. This ensures that the volume is placed on the same node as the workload.

For our example, we create a simple Pod, which mounts or volume by referencing the PersistentVolumeClaim. .my-first-linstor-volume-pod.yaml

You can create the Pod with the following command:

Running kubectl describe pod fedora can be used to confirm that Pod scheduling and volume attachment succeeded. Examining the PersistentVolumeClaim, we can see that it is now bound to a volume.

To remove a volume, verify that no pod is using it and then delete the PersistentVolumeClaim using the kubectl command. For example, to remove the volume that we just made, run the following two commands, noting that the Pod must be unscheduled before the PersistentVolumeClaim will be removed:

The following storage class contains all currently available parameters to configure the provisioned storage.

The csi.storage.k8s.io/fstype parameter sets the file system type to create for volumeMode: FileSystem PVCs. Currently supported are: * ext4 (default) * xfs ==== autoPlace

autoPlace is an integer that determines the amount of replicas a volume of this StorageClass will have. For instance, autoPlace: "3" will produce volumes with three-way replication. If neither autoPlace nor nodeList are set, volumes will be automatically placed on one node. IMPORTANT: If you use this option, you must not use nodeList. IMPORTANT: You have to use quotes, otherwise Kubernetes will complain about a malformed StorageClass. TIP: This option (and all options which affect auto-placement behavior) modifies the number of LINSTOR nodes on which the underlying storage for volumes will be provisioned and is orthogonal to which kubelets those volumes will be accessible from.

placementCount is an alias for autoPlace

The LINSTOR Resource Group (RG) to associate with this StorageClass. If not set, a new RG will be created for each new PVC. ==== storagePool

storagePool is the name of the LINSTOR storage pool that will be used to provide storage to the newly-created volumes. CAUTION: Only nodes configured with this same storage pool with be considered for auto-placement. Likewise, for StorageClasses using nodeList all nodes specified in that list must have this storage pool configured on them. ==== disklessStoragePool

disklessStoragePool is an optional parameter that only affects LINSTOR volumes that are assigned as “diskless” to kubelets, that is, as clients. If you have a custom diskless storage pool defined in LINSTOR, you will specify that here.

A comma-separated list of layers to use for the created volumes. The available layers and their order are described towards the end of this section. Defaults to drbd,storage ==== placementPolicy

Select from one of the available volume schedulers: * AutoPlaceTopology, the default: Use topology information from Kubernetes together with user provided constraints (see [s-kubernetes-replicasonsame] and [s-kubernetes-replicasondifferent]). * AutoPlace Use the LINSTOR auto-placement feature, influenced by [s-kubernetes-replicasonsame] and [s-kubernetes-replicasondifferent] * FollowTopology: Use CSI Topology information to place at least one volume in each “preferred” zone. Only usable if CSI Topology is enabled. * Manual: Use only the nodes listed in nodeList and clientList. * Balanced: EXPERIMENTAL Place volumes across failure domains, using the least used storage pool on each selected node. ==== allowRemoteVolumeAccess

Control on which nodes a volume is accessible. The value for this option can take two different forms:

encryption is an optional parameter that determines whether to encrypt volumes. LINSTOR must be configured for encryption for this to work properly. ==== nodeList

nodeList is a list of nodes for volumes to be assigned to. This will assign the volume to each node and it will be replicated among all of them. This can also be used to select a single node by hostname, but it’s more flexible to use replicasOnSame to select a single node. IMPORTANT: If you use this option, you must not use autoPlace. TIP: This option determines on which LINSTOR nodes the underlying storage for volumes will be provisioned and is orthogonal from which kubelets these volumes will be accessible.

clientList is a list of nodes for diskless volumes to be assigned to. Use in conjunction with [s-kubernetes-nodelist]. ==== replicasOnSame

replicasOnDifferent takes a list of properties to consider, same as replicasOnSame. There are two modes of using replicasOnDifferent: * Preventing volume placement on specific nodes: + If a value is given for the property, the nodes which have that property-value pair assigned will be considered last. + Example: replicasOnDifferent: "no-csi-volumes=true" will place no volume on any node with property no-csi-volumes=true unless there are not enough other nodes to fulfill the autoPlace setting. * Distribute volumes across nodes with different values for the same key: + If no property value is given, LINSTOR will place the volumes across nodes with different values for that property if possible. + Example: Assuming there are 4 nodes, node-a1 and node-a2 are configured with zone=a. node-b1 and node-b2 are configured with zone=b. Using a StorageClass with autoPlace: "2" and replicasOnDifferent: "zone", LINSTOR will create one replica on either node-a1 or node-a2 and one replica on either node-b1 or node-b2.

Create a diskless resource on all nodes that were not assigned a diskful resource.

Do not place the resource on a node which has a resource with a name matching the regular expression. ==== fsOpts fsOpts is an optional parameter that passes options to the volume’s file system at creation time. IMPORTANT: These values are specific to your chosen file system. ==== mountOpts mountOpts is an optional parameter that passes options to the volume’s file system at mount time.

Extra arguments to pass to xfs_io, which gets called before right before first use of the volume. ==== property.linstor.csi.linbit.com/*

Parameters starting with property.linstor.csi.linbit.com/ are translated to LINSTOR properties that are set on the Resource Group associated with the StorageClass.

For example, to set DrbdOptions/auto-quorum to disabled, use:

The full list of options is available here

Snapshots create a copy of the volume content at a particular point in time. This copy remains untouched when you make modifications to the volume content. This, for example, enables you to create backups of your data before performing modifications or deletions on your data. Because a backup is useless unless you have a way to restore it, this section describes how to create a snapshot, and how to restore it, for example, in the case of accidental deletion of your data. The next subsection contains instructions around snapshots within Operator v2 deployments. If you have deployed LINBIT SDS in Kubernetes by using Operator v1, skip ahead to the [s-kubernetes-add-snaphot-support-v1] subsection. ==== Working With Snapshots

Before you can add snapshot support within a LINBIT SDS deployment, you need to meet the following environment prerequisites:

LINSTOR can store snapshots on S3 compatible storage for disaster recovery. This is integrated in Kubernetes using a special VolumeSnapshotClass:

Check the LINSTOR snapshot guide on the exact meaning of the snap.linstor.csi.linbit.com/ parameters. The credentials used to log in are stored in a separate secret, as show in the example above.

Referencing the above storage class when creating snapshots causes the snapshots to be automatically uploaded to the configured S3 storage.

The following example show common scenarios where you want to optimize volume accessibility and locality. It also includes examples of how to spread volume replicas across zones in a cluster.

In some circumstances it might be necessary to disable the DRBD module loader entirely. For example, if you are using an immutable operating system, and DRBD and other modules are loaded as part of the host configuration.

To disable the DRBD module loader completely, apply the following YAML configuration to your deployment:

By default, the Operator will try to find a DRBD module loader that matches the host operating system. The Operator determines the host distribution by inspecting the .status.nodeInfo.osImage field of the Kubernetes Node resource. A user-defined image can be used if the automatic mapping does not succeed or if you have different module loading requirements.

The following YAML configuration overrides the chosen DRBD module loader image with a user-defined image example.com/drbd-loader:v9:

drbd.io, available to LINBIT customers only, maintains the following module loader container images:

If you need to create a module loader image for your own distribution, you can refer to the container source files which are available in the upstream Piraeus project.

By default, the DRBD module loader will try to build the kernel module from source. The module loader can also be configured to load the module from a DEB or RPM package included in the image, or skip loading DRBD entirely.

To change the behavior of the DRBD module loader, set the LB_HOW environment variable to an appropriate value shown in the following table:

After setting the LB_HOW environment variable, apply the following YAML configuration to your deployment. Based on the name within the metadata section, the example below would be used with an LB_HOW environment variable that was set to deps_only.

Switching from the default container network to the host network for DRBD replication is possible at any time. Existing DRBD resources will then be reconfigured to use the host network interface.

To configure the host network for the LINSTOR satellite, apply the following YAML configuration to your deployment:

After the satellite pods are recreated, they will use the host network. Any existing DRBD resources are reconfigured to use a new IP address on the host network rather than an IP address on the container network.

Switching back from host network to container network involves manually resetting the configured peer addresses used by DRBD. You can do this by rebooting every node, or by manually resetting the addresses by using the drbdadm CLI command on each node. Each method is described below.

[[s-kubernetes-drbd-replication-switching-from-host-to-container-network-node-rebooting-v2 ===== Rebooting Nodes to Switch DRBD Replication from the Host to the Container Network

First, you need to remove the LinstorSatelliteConfiguration that set hostNetwork: true. You can do this by entering the following kubectl command:

Next, reboot each cluster node, either serially, one by one, or else all at once. In general, replication will not work between rebooted nodes and non-rebooted nodes. The non-rebooted nodes will continue to use the host network addresses, which are generally not reachable from the container network.

After all nodes have restarted, all resources will be configured to use the container network, and all DRBD connections should be connected again.

[[s-kubernetes-drbd-replication-switching-from-host-to-container-network-node-drbdadm-v2 ===== Using the DRBD Administration Tool to Switch DRBD Replication from the Host to the Container Network

First, you need to temporarily stop all DRBD replication and suspend all DRBD volume I/O operations by using the drbdadm suspend-io all command. Enter the command once on each LINSTOR satellite pod.

Next, disconnect all DRBD connections on all nodes.

Next, you can safely reset all DRBD connection paths. This frees the connection on each node to be moved to the container network.

Finally, remove the LinstorSatelliteConfiguration resource configuration that set hostNetwork: true. This will result in the creation of new LINSTOR satellite pods that use the container network.

After the pods are recreated and the LINSTOR satellites are Online, the DRBD resource will be reconfigured and resume I/O operations.

In Operator v1 deployments, the operator will set up monitoring containers along the existing components and make them available as a Service.

If you use the Prometheus Operator, the LINSTOR Operator will also set up the ServiceMonitor instances. The metrics will automatically be collected by the Prometheus instance associated to the operator, assuming watching the Piraeus namespace is enabled.

To disable exporting of metrics, set operator.satelliteSet.monitoringImage to an empty value.

If you deployed LINSTOR using Operator v2, you can upgrade LINBIT SDS by deploying the newer version of LINSTOR Operator.

To upgrade, change the tag referenced in the kustomization file:

Then, apply the kustomization.yaml file, by using the following kubectl command, and wait for the upgrade to complete:

Before upgrading to a new release, you should ensure you have an up-to-date backup of the LINSTOR database. If you are using the etcd database packaged in the LINSTOR Chart, see here

Upgrades will update to new versions of the following components:

Some versions require special steps, refer to instructions here The main command to upgrade to a new LINSTOR Operator version is:

If you used any customizations on the initial install, pass the same options to helm upgrade. The options currently in use can be retrieved from Helm.

This triggers the rollout of new pods. After a short wait, all pods should be running and ready. Check that no errors are listed in the status section of LinstorControllers, LinstorSatelliteSets and LinstorCSIDrivers.

The LINSTOR Operator will create a backup of the database before upgrading. This makes it possible to roll back to a known state should something go wrong during the upgrade.

There are situations in which the Operator can’t create the backup automatically. This might be because:

Some versions require special steps, see below.

The next steps are exactly the same as in Kubernetes:

The LINSTOR container images come with the LINBIT GUI preinstalled. To expose it on your cluster, configure an OpenShift Route resource:

The GUI is now accessible at https://linstor-op-cs-storage.apps.oc.example.com/ui/

OpenShift includes a fully configured monitoring stack. While most of the monitoring stack is only intended for OpenStack infrastructure, basic monitoring for LINBIT SDS is possible.

First, ensure that monitoring of user-defined projects is enabled in OpenShift by following the steps in Red Hat documentation.

Once monitoring for user-defined projects is enabled, the LINSTOR Operator automatically detects the presence of OpenShift’s Prometheus-based monitoring stack and configures the ServiceMonitor resources. The Prometheus instance will scrape metrics for DRBD and LINSTOR from all Cluster nodes.

To run LINSTOR, every Nomad agent needs to be configured to:

The LINSTOR Controller is deployed as a service with no replicas. At any point in time, there can only be one LINSTOR Controller running in a cluster. It is possible to restart the controller on a new node, provided it still has access to the database. See Creating a Highly Available LINSTOR Cluster for more information.

The following example will create a Nomad job starting a single LINSTOR Controller in datacenter dc1 and connect to an external database.

Apply the job by running:

In Nomad, the LINSTOR satellites are deployed as a system job that runs in a privileged container. In addition to the satellites, the job will also load the DRBD module along with other kernel modules used by LINSTOR.

The following example will create a Nomad job starting a LINSTOR satellite on every node in data center dc1.

Apply the job by running:

Once the linstor-controller and linstor-satellite jobs are running, you can start configuring the cluster using the linstor command line tool.

This can done:

In all cases, you need to add the satellites to your cluster and create some storage pools. For example, to add the node nomad-01.example.com and configure a LVM Thin storage pool, you would run:

To use the volume in a job, add the volume and volume_mount stanzas to the job specification:

LINSTOR can create snapshots of existing volumes, provided the underlying storage pool driver supports snapshots.

The following command creates a snapshot named snap1 of the volume vol1.

You can use a snapshot to pre-populate an existing volume with data from the snapshot

Version 7 of the plugin uses a LINSTOR controller API that is available from LINSTOR version 1.21.1 onwards. Make sure that your LINSTOR setup (controller and satellites) use at least that version.

Version 5 of the plugin drops compatibility with the legacy configuration options “storagepool” and “redundancy”. Version 5 requires a “resourcegroup” option, and obviously a LINSTOR resource group. The old options should be removed from the configuration.

Configuring LINSTOR is described in Section Configuring LINSTOR, a typical example follows:

Let’s assume the pool was set to “mypool”, and redundancy to 3.

Version 6.0.0 of the plugin drops all code related to the redundancy setting. This is handled by LINSTOR resource groups (resourcegroup setting) for a very long time. No change should be required.

The controllervm setting, which was intended for executing a LINSTOR controller in a VM manged by LINSTOR is gone. Using drbd-reactor to realize a highly available LINSTOR controller is what we suggest.

The settings statuscache and preferlocal are now enabled by default.

With version 6 PVE added additional parameters to some functions and rightfully reset their “APIAGE”. This means that old plugins, while actually usable as they don’t use any of these changed functions do not work anymore. Please upgrade to plugin version 5.2.1 at least.

To use LINSTOR in Proxmox, you will need to install the DRBD kernel module. The DRBD 9 kernel module is installed as a kernel module source package (drbd-dkms). Therefore, you will have to install the Proxmox VE kernel headers package, pve-headers, before you install the DRBD kernel module from LINBIT’s repositories. Following that order ensures that the kernel module will build properly for your kernel.

If you do not plan to install the latest Proxmox kernel, you have to install kernel headers matching your current running kernel (for example, pve-headers-$(uname -r)). If you missed this step, then you can still rebuild the drbd-dkms package against your current kernel (so long as you have installed kernel headers in advance) by entering the apt install --reinstall drbd-dkms command.

If you are a LINBIT customer, or you have an evaluation account, you can enable LINBIT’s drbd-9 repository on your node and then update your repositories by using an apt update command.

You can then install the DRBD kernel module, DRBD utilities, and the LINSTOR Proxmox plugin by entering:

LINBIT provides a dedicated public repository for Proxmox VE users. This repository not only contains the Proxmox plugin, but the whole DRBD SDS stack including a DRBD SDS kernel module and user-space utilities.

You can add LINBIT’s public repository by entering the commands below, setting $PVERS to your Proxmox VE major version (for example, “7”, not “7.1”):

After adding the LINBIT package repository, you can install the Proxmox plugin and other necessary components (DRBD kernel module and utilities), by entering the following command: