Objects are the end result which LINSTOR presents to the end-user or application, such as: Kubernetes/OpenShift, a replicated block device (DRBD), NVMeOF target, and others.

To use LINSTOR to create storage volumes, you will need to install a volume manager, either LVM or ZFS, if one is not already installed on your system.

If you are a LINBIT customer, you can download a LINBIT created helper script and run it on your nodes to:

Enabling LINBIT package repositories will give you access to LINBT software packages, DRBD kernel modules, and other related software such as cluster managers and OCF scripts. You can then use a package manager to fetch, install, and manage updating installed packages.

After registering your node and enabling the drbd-9 LINBIT package repository, you can use a DEB, RPM, or YaST2 based package manager to install LINSTOR and related components.

The LINSTOR project’s GitHub page is here: https://github.com/LINBIT/linstor-server.

LINBIT also has downloadable archived files of source code for LINSTOR, DRBD, and more, available here: https://linbit.com/linbit-software-download-page-for-linstor-and-drbd-linux-driver/.

In clusters where you have only one kind of storage and the capability to hot swap storage devices, you may choose a model where you create one storage pool per physical backing device. The advantage of this model is to confine failure domains to a single storage device.

Both the Exos and LVM2 storage providers offer the option of multiple server nodes directly connected to the storage array and drives. With LVM2 the external locking service (lvmlockd) manages volume groups created with the –shared options with vgcreate. The --shared-space can be used when configuring a LINSTOR pool to use the same LVM2 volume group accessible by two or more nodes. The example below shows using the LVM2 volume group UUID as the shared space identifier for a pool accessible by nodes alpha and bravo:

Exos pools will use the Exos pool serial number by default for the shared-space identifier.

Since linstor-server 1.5.2 and a recent linstor-client, LINSTOR can create LVM/ZFS pools on a satellite for you. The linstor-client has the following commands to list possible disks and create storage pools, but such LVM/ZFS pools are not managed by LINSTOR and there is no delete command, so such action must be done manually on the nodes.

Will give you a list of available disks grouped by size and rotational(SSD/Magnetic Disk).

It will only show disks that pass the following filters:

With the create-device-pool command you can create a LVM pool on a disk and also directly add it as a storage-pool in LINSTOR.

If the --storage-pool option was provided, LINSTOR will create a storage-pool with the given name.

For more options and exact command usage please check the linstor-client help.

LINSTOR has the following supported storage plug-ins as of writing:

With the resource create command you may assign a resource definition to named nodes explicitly.

It is possible to have LINSTOR select nodes and storage pools to deploy a resource to, by using the linstor resource create --auto-place or linstor resource-definition auto-place commmands. This section will use the resource create --auto-place command in examples. However, you can use either command to produce the same results.

In the following example, the value after the --auto-place option tells LINSTOR how many replicas you want to have. The storage-pool option should be obvious.

Maybe not so obvious is that you may omit the --storage-pool option, then LINSTOR may select a storage pool on its own. The selection follows these rules:

The remaining storage pools will be rated by different strategies. LINSTOR has currently three strategies:

The scores of the strategies will be normalized, weighted and summed up, where the scores of minimizing strategies will be converted first to allow an overall maximization of the resulting score.

The weights of the strategies can be configured with the following command:

The strategy names are listed above and the weight can be an arbitrary decimal.

Finally, LINSTOR tries to find the best matching group of storage pools meeting all requirements. This step also considers other auto-placement restrictions such as --replicas-on-same, --replicas-on-different, --do-not-place-with, --do-not-place-with-regex, --layer-list, and --providers.

You can delete a resource definition by using the command:

This will remove the named resource definition from the entire LINSTOR cluster. The resource is removed from all nodes and the resource entry is marked for removal from LINSTOR’s database tables. After LINSTOR has removed the resource from all the nodes, the resource entry is removed from LINSTOR’s database tables.

You can delete a resource using the command:

Unlike deleting a resource definition, this command will only delete a LINSTOR resource from the node (or nodes) that you specify. The resource is removed from the node and the resource entry is marked for removal from LINSTOR’s database tables. After LINSTOR has removed the resource from the node, the resource entry is removed from LINSTOR’s database tables.

Deleting a LINSTOR resource may have implications for a cluster, beyond just removing the resource. For example, if the resource is backed by a DRBD layer, removing a resource from one node in a three node cluster could also remove certain quorum related DRBD options, if any existed for the resource. After removing such a resource from a node in a three node cluster, the resource would no longer have quorum as it would now only be deployed on two nodes in the three node cluster.

After running a linstor resource delete command to remove a resource from a single node, you might see informational messages such as:

Also unlike deleting a resource definition, you can delete a resource while there are existing snapshots of the resource’s storage pool. Any existing snapshots for the resource’s storage pool will persist.

You can delete a resource group by using the command:

As you might expect, this command deletes the named resource group. You can only delete a resource group if it has no associated resource definitions, otherwise LINSTOR will present an error message, such as:

To resolve this error so that you can delete the resource group, you can either delete the associated resource definitions, or your can move the resource definitions to another (existing) resource group:

You can find which resource definitions are associated with your resource group by entering the following command:

For configuring the highly-available storage we use LINSTOR itself. This has the advantage that the storage is under LINSTOR control and can for example be easily extended to new cluster nodes. Just create a new resource with 200MB in size. This could look like this, you certainly need to adapt the storage pool name:

From now on we assume the resource’s name is “linstor_db”. It is crucial that your cluster qualifies for auto-quorum and uses the io-error policy (see Section Auto-Quorum Policies), and that auto-promote is disabled.

After the resource is created, it is time to move the LINSTOR DB to the new storage and to create a systemd mount service. First we stop the current controller and disable it, as it will be managed by drbd-reactor later.

Copy the /etc/systemd/system/var-lib-linstor.mount mount file to all the standby nodes for the linstor controller. Again, do not systemctl enable any of these services, they get managed by drbd-reactor.

The next step is to install LINSTOR controllers on all nodes that have access to the linstor_db DRBD resource (as they need to mount the DRBD volume) and which you want to become a possible LINSTOR controller. It is important that the controllers are manged by drbd-reactor, so verify that the linstor-controller.service is disabled on all nodes! To be sure, execute systemctl disable linstor-controller on all cluster nodes and systemctl stop linstor-controller on all nodes except the one it is currently running from the previous step. Also verify that you have set chattr +i /var/lib/linstor on all potential controller nodes if you use LINSTOR version equal or greater to 1.14.0.

For starting and stopping the mount service and the linstor-controller service we use drbd-reactor. Install this component on all nodes that could become a LINSTOR controller and edit their /etc/drbd-reactor.d/linstor_db.toml configuration file. It should contain an enabled promoter plug-in section like this:

Depending on your requirements you might also want to set an on-stop-failure action and set stop-services-on-exit.

After that restart drbd-reactor and enable it on all the nodes you configured it.

Check that there are no warnings from drbd-reactor service in the logs by running systemctl status drbd-reactor. As there is already an active LINSTOR controller things will just stay the way they are. Run drbd-reactorctl status linstor_db to check the health of the linstor_db target unit.

The last but nevertheless important step is to configure the LINSTOR satellite services to not delete (and then regenerate) the resource file for the LINSTOR controller DB at its startup. Do not edit the service files directly, but use systemctl edit. Edit the service file on all nodes that could become a LINSTOR controller and that are also LINSTOR satellites.

After this change you should execute systemctl restart linstor-satellite on all satellite nodes.

NVMe-oF/NVMe-TCP allows LINSTOR to connect diskless resources to a node with the same resource where the data is stored over NVMe fabrics. This leads to the advantage that resources can be mounted without using local storage by accessing the data over the network. LINSTOR is not using DRBD in this case, and therefore NVMe resources provisioned by LINSTOR are not replicated, the data is stored on one node.

To use NVMe-oF/NVMe-TCP with LINSTOR the package nvme-cli needs to be installed on every Node which acts as a Satellite and will use NVMe-oF/NVMe-TCP for a resource:

To make a resource which uses NVMe-oF/NVMe-TCP an additional parameter has to be given as you create the resource-definition:

To use NVMe-TCP rather than the default NVMe-oF, the following property needs to be set:

The property NVMe/TRType can alternatively be set on resource-group or controller level.

Next, create the volume-definition for our resource:

Before you create the resource on your nodes you have to know where the data will be stored locally and which node accesses it over the network.

First we create the resource on the node where our data will be stored:

On the nodes where the resource-data will be accessed over the network, the resource has to be defined as diskless:

Now you can mount the resource nvmedata on one of your nodes.

Since version 1.5.0 the additional Layer openflex can be used in LINSTOR. From LINSTOR’s perspective, the OpenFlex Composable Infrastructure takes the role of a combined layer acting as a storage layer (like LVM) and also providing the allocated space as an NVMe target. OpenFlex has a REST API which is also used by LINSTOR to operate with.

As OpenFlex combines concepts of LINSTOR’s storage as well as NVMe-layer, LINSTOR was added both, a new storage driver for the storage pools as well as a dedicated openflex layer which uses the mentioned REST API.

In order for LINSTOR to communicate with the OpenFlex-API, LINSTOR needs some additional properties, which can be set once on controller level to take LINSTOR-cluster wide effect:

Once that is configured, we can now create LINSTOR objects to represent the OpenFlex architecture. The theoretical mapping of LINSTOR objects to OpenFlex objects are as follows: Obviously an OpenFlex storage pool is represented by a LINSTOR storage pool. As the next thing above a LINSTOR storage pool is already the node, a LINSTOR node represents an OpenFlex storage device. The OpenFlex objects above storage device are not mapped by LINSTOR.

When using NVMe, LINSTOR was designed to run on both sides, the NVMe target as well as on the NVMe initiator side. In the case of OpenFlex, LINSTOR cannot (or even should not) run on the NVMe target side as that is completely managed by OpenFlex. As LINSTOR still needs nodes and storage pools to represent the OpenFlex counterparts, the LINSTOR client was extended with special node create commands since 1.0.14. These commands not only accept additionally needed configuration data, but also starts a “special satellite” besides the already running controller instance. This special satellites are completely LINSTOR managed, they will shutdown when the controller shuts down and will be started again when the controller starts.

The new client command for creating a “special satellite” representing an OpenFlex storage device is:

The arguments are as follows:

The last step of the configuration is the creation of LINSTOR storage pools:

Once all necessary storage pools are created in LINSTOR, the next steps are similar to the usage of using an NVMe resource with LINSTOR. Here is a complete example:

A DM-Writecache device is composed of two devices: one storage device and one cache device. LINSTOR can setup such a writecache device, but needs some additional information, like the storage pool and the size of the cache device.

The two properties set in the examples are mandatory, but can also be set on controller level which would act as a default for all resources with WRITECACHE in their --layer-list. However, please note that the Writecache/PoolName refers to the corresponding node. If the node does not have a storage-pool named pmempool you will get an error message.

The 4 mandatory parameters required by DM-Writecache are either configured through a property or figured out by LINSTOR. The optional properties listed in the mentioned link can also be set through a property. Please see linstor controller set-property --help for a list of Writecache/* property-keys.

Using --layer-list DRBD,WRITECACHE,STORAGE while having DRBD configured to use external metadata, only the backing device will use a writecache, not the device holding the external metadata.

LINSTOR can also setup a DM-Cache device, which is very similar to the DM-Writecache from the previous section. The major difference is that a cache device is composed by three devices: one storage device, one cache device and one meta device. The LINSTOR properties are quite similar to those of the writecache but are located in the Cache namespace:

Please see linstor controller set-property --help for a list of Cache/* property-keys and default values for omitted properties.

Using --layer-list DRBD,CACHE,STORAGE while having DRBD configured to use external metadata, only the backing device will use a cache, not the device holding the external metadata.

The storage layer will provide new devices from well known volume managers like LVM, ZFS or others. Every layer combination needs to be based on a storage layer, even if the resource should be diskless – for that type there is a dedicated diskless provider type.

For a list of providers with their properties please see Storage Providers.

For some storage providers LINSTOR has special properties:

A storage pool with the type remote SPDK can only be created on a “special satellite”. For this you first need to start a new satellite using the command:

This will start a new satellite instance running on the same machine as the controller. This special satellite will do all the REST based RPC communication towards the remote SPDK proxy. As the help message of the LINSTOR command shows, the administrator might want to use additional settings when creating this special satellite:

The difference between the --api-* and their corresponding --api-\*-env versions is that the version with the -env ending will look for an environment variable containing the actual value to use whereas the --api-\* version directly take the value which is stored in the LINSTOR property. Administrators might not want to save the --api-pw in plain text, which would be clearly visible using commands like linstor node list-property <nodeName>.

Once that special satellite is up and running the actual storage pool can be created:

Whereas node_name is self-explanatory, name is the name of the LINSTOR storage pool and driver_pool_name refers to the SPDK logical volume store.

Once this remotespdk storage pool is created the remaining procedure is quite similar as using NVMe: First the target has to be created by creating a simple “diskful” resource followed by a second resource having the --nvme-initiator option enabled.

To use multiple network paths for DRBD setups, the PrefNic property is not sufficient. Instead the linstor node interface and linstor resource-connection path commands should be used, as shown below.

In the example above we define two network interfaces (nic1 and nic2) for each node. The last two commands create network path entries in the generated DRBD .res file. This is the relevant part of the resulting .res file:

Below are details about the commands.

Before LINSTOR can encrypt any volume a master passphrase needs to be created. This can be done with the linstor-client.

crypt-create-passphrase will wait for the user to input the initial master passphrase (as all other crypt commands will with no arguments).

If you ever want to change the master passphrase this can be done with:

The luks layer can be added when creating the resource-definition or the resource itself, whereas the former method is recommended since it will be automatically applied to all resource created from that resource-definition.

To enter the master passphrase (after controller restart) use the following command:

It is possible to automate the process of creating and re-entering the master passphrase.

To use this, either an environment variable called MASTER_PASSPHRASE or an entry in /etc/linstor/linstor.toml containing the master passphrase has to be created.

The required linstor.toml looks like this:

If either one of these is set, then every time the controller starts it will check whether a master passphrase already exists. If there is none, it will create a new master passphrase as specified. Otherwise, the controller enters the passphrase.

Some evacuation cases may need special planning. For example, if you are evacuating more than one node, you can exclude the nodes from participating in LINSTOR’s resource autoplacer. You can do this by using the following command on each node that you want to evacuate:

This ensures that LINSTOR will not place resources from a node that you are evacuating onto another node that you plan on evacuating.

If you already ran a node evacuate command that has either completed or still has resources in an “Evacuating” state, you can remove the “Evacuating” state from a node by using the node restore command. This will work so long as you have not yet run a node delete command.

After restoring the node, you should use the node set-property <node_name> AutoplaceTarget true command, if you previously set the AutoplaceTarget property to “false”. This way, LINSTOR can again place resources onto the node automatically, to fulfill placement count properties that you might have set for resources in your cluster.

Assuming a resource definition named ‘resource1’ which has been placed on some nodes, a snapshot can be created as follows:

This will create snapshots on all nodes where the resource is present. LINSTOR will ensure that consistent snapshots are taken even when the resource is in active use.

Setting the resource-definition property AutoSnapshot/RunEvery LINSTOR will automatically create snapshots every X minute. The optional property AutoSnapshot/Keep can be used to clean-up old snapshots which were created automatically. No manually created snapshot will be cleaned-up / deleted. If AutoSnapshot/Keep is omitted (or ⇐ 0), LINSTOR will keep the last 10 snapshots by default.

The following steps restore a snapshot to a new resource. This is possible even when the original resource has been removed from the nodes where the snapshots were taken.

First define the new resource with volumes matching those from the snapshot:

At this point, additional configuration can be applied if necessary. Then, when ready, create resources based on the snapshots:

This will place the new resource on all nodes where the snapshot is present. The nodes on which to place the resource can also be selected explicitly; see the help (linstor snapshot resource restore -h).

LINSTOR can roll a resource back to a snapshot state. The resource must not be in use. That is, it may not be mounted on any nodes. If the resource is in use, consider whether you can achieve your goal by restoring the snapshot instead.

Rollback is performed as follows:

A resource can only be rolled back to the most recent snapshot. To roll back to an older snapshot, first delete the intermediate snapshots.

An existing snapshot can be removed as follows:

Snapshots can be shipped between LINSTOR nodes or between different LINSTOR clusters, as well as to an S3 storage such as Amazon S3 or min.io.

The following tools need to be installed on the satellites that are going to send or receive snapshots:

You create a backup shipping schedule by using the LINSTOR client schedule create command and defining the frequency of backup shipping using cron syntax. You also need to set options that name the schedule and define various aspects of the backup shipping, such as on-failure actions, the number of local and remote backup copies to keep, and whether to also schedule incremental backup shipping.

At a minimum, the command needs a schedule name and a full backup cron schema to create a backup shipping schedule. An example command would look like this:

You can modify a backup shipping schedule by using the LINSTOR client schedule modify command. The syntax for the command is the same as that for the schedule create command. The name that you specify with the schedule modify command must be an already existing backup schedule. Any options to the command that you do not specify will retain their existing values. If you want to set the keep-local or keep-remote options back to their default values, you can set them to “all”. If you want to set the max-retries option to its default value, you can set it to “forever”.

Your physical storage is not infinite and your remote storage has a cost, so you will likely want to set limits on the number of snapshots and backups you keep.

Both the --keep-remote and --keep-local options deserve special mention as they have implications beyond what may be obvious. Using these options, you specify how many snapshots or full backups should be kept, either on the local source or the remote destination.

You can list your backup shipping schedules by using the linstor schedule list command.

For example:

The LINSTOR client schedule delete command completely deletes a backup shipping schedule LINSTOR object. The command’s only argument is the schedule name that you want to delete. If the deleted schedule is currently creating or shipping a backup, the scheduled shipping process is stopped. Depending on at which point the process stops, a snapshot, or a backup, or both, might not be created and shipped.

This command does not affect previously created snapshots or successfully shipped backups. These will be retained until they are manually deleted.

You can use the LINSTOR client backup schedule enable command to enable a previously created backup shipping schedule. The command has the following syntax:

To disable a previously enabled backup shipping schedule, you use the LINSTOR client backup schedule disable command. The command has the following syntax:

If you include the option specifying either a resource group or resource definition, as described in the backup schedule enable command example above, then you disable the schedule only for that resource group or resource definition.

For example, if you omitted specifying a resource group or resource definition in an earlier backup schedule enable command, LINSTOR would schedule backup shipping for all its deployed resources that can make snapshots. Your disable command would then only affect the resource group or resource definition that you specify with the command. The backup shipping schedule would still apply to any deployed LINSTOR resources besides the specified resource group or resource definition.

The same as for the backup schedule enable command, if you specify neither a resource group nor a resource definition, then LINSTOR disables the backup shipping schedule at the controller level for all deployed LINSTOR resources.

You can use the linstor backup schedule delete command to granularly delete either a specified resource definition or a resource group from a backup shipping schedule, without deleting the schedule itself. This command has the same syntax and arguments as the backup schedule disable command. If you specify neither a resource group nor a resource definition, the backup shipping schedule you specify will be deleted at the controller level.

It may be helpful to think about the backup schedule delete command as a way that you can remove a backup shipping schedule-remote pair from a specified LINSTOR object level, either a resource definition, a resource group, or at the controller level if neither is specified.

The backup schedule delete command does not affect previously created snapshots or successfully shipped backups. These will be retained until they are manually deleted, or until they are removed by the effects of a still applicable keep-local or keep-remote option.

You might want to use this command when you have disabled a backup schedule for multiple LINSTOR object levels and later want to affect a granular change, where a backup schedule enable command might have unintended consequences.

For example, consider a scenario where you have a backup schedule-remote pair that you enabled at a controller level. This controller has a resource group, myresgroup that has several resource definitions, resdef1 through resdef9, under it. For maintenance reasons perhaps, you disable the schedule for two resource definitions, resdef1 and resdef2. You then realize that further maintenance requires that you disable the backup shipping schedule at the resource group level, for your myresgroup resource group.

After completing some maintenance, you are able to enable the backup shipping schedule for resdef3 through resdef9, but you are not yet ready to resume (enable) backup shipping for resdef1 and resdef2. You can enable backup shipping for each resource definition individually, resdef3 through resdef9, or you can use the backup schedule delete command to delete the backup shipping schedule from the resource group, myresgroup. If you use the backup schedule delete command, backups of resdef3 through resdef9 will ship again because the backup shipping schedule is enabled at the controller level, but resdef1 and resdef2 will not ship because the backup shipping schedule is still disabled for them at the resource definition level.

When you complete your maintenance and are again ready to ship backups for resdef1 and resdef2, you can delete the backup shipping schedule for those two resource definitions to return to your starting state: backup shipping scheduled for all LINSTOR deployed resources at the controller level. To visualize this it may be helpful to refer to the decision tree diagram for how LINSTOR decides whether or not to ship a backup in the How the LINSTOR Controller Determines Scheduled Backup Shipping subsection.

You can list backup schedules by resource, using the LINSTOR client schedule list-by-resource command. This command will show LINSTOR resources and how any backup shipping schedules apply and to which remotes they are being shipped. If resources are not being shipped then the command will show:

If resources have schedule-remote-pairs and are being shipped, the command output will show when the last backup was shipped and when the next backup is scheduled to ship. It will also show whether the next and last backup shipments were full or incremental backups. Finally, the command will show when the next planned incremental (if any) and full backup shipping will occur.

You can use the --active-only flag with the schedule list-by-resource command to filter out all resources that are not being shipped.

To determine if the LINSTOR Controller will ship a deployed LINSTOR resource with a certain backup schedule for a given remote destination, the LINSTOR Controller uses the following logic:

As the diagram shows, enabled or disabled backup shipping schedules have effect in the following order:

A backup shipping schedule-remote pair that is enabled or disabled at a preceding level will override the enabled or disabled status for the same schedule-remote pair at a later level.

To determine how a LINSTOR resource will be affected by scheduled backup shipping, you can use the LINSTOR client schedule list-by-resource-details command for a specified LINSTOR resource.

The command will output a table that shows on what LINSTOR object level a backup shipping schedule is either not set (empty cell), enabled, or disabled.

By using this command, you can determine on which level you need to make a change to enable, disable, or delete scheduled backup shipping for a resource.

Example output could look like this:

To move a resource between nodes without reducing redundancy at any point, LINSTOR’s disk migrate feature can be used. First create a diskless resource on the target node, and then add a disk using the --migrate-from option. This will wait until the data has been synced to the new disk and then remove the source disk.

For example, to migrate a resource backups from ‘alpha’ to ‘bravo’:

LINSTOR can also be configured to automatically enable the above mentioned Proxy connection between two nodes. For this automation, LINSTOR first needs to know on which site each node is.

As the Site property might also be used for other site-based decisions in future features, the DrbdProxy/AutoEnable also has to be set to true:

This property can also be set on node, resource-definition, resource and resource-connection level (from left to right in increasing priority, whereas the controller is the left-most, that is, the least prioritized level).

Once this initialization steps are completed, every newly created resource will automatically check if it has to enable DRBD proxy to any of its peer-resources.

A sample PostgreSQL linstor.toml looks like this:

A sample MariaDB linstor.toml looks like this:

etcd is a distributed key-value store that makes it easy to keep your LINSTOR database distributed in a HA-setup. The etcd driver is already included in the LINSTOR-controller package and only needs to be configured in the linstor.toml.

More information about how to install and configure etcd can be found here: etcd docs

And here is a sample [db] section from the linstor.toml:

To make LINSTOR’s administrative tasks more accessible and also available for web-frontends a REST API has been created. The REST API is embedded in the linstor-controller and since LINSTOR 0.9.13 configured through the linstor.toml configuration file.

If you want to use the REST API the current documentation can be found on the following link: https://app.swaggerhub.com/apis-docs/Linstor/Linstor/

The HTTP REST API can also run secured by HTTPS and is highly recommended if you use any features that require authorization. To do so you have to create a Java keystore file with a valid certificate that will be used to encrypt all HTTPS traffic.

Here is a simple example on how you can create a self signed certificate with the keytool that is included in the Java Runtime:

keytool will ask for a password to secure the generated keystore file and is needed for the LINSTOR Controller configuration. In your linstor.toml file you have to add the following section:

Now (re)start the linstor-controller and the HTTPS REST API should be available on port 3371.

More information about how to import other certificates can be found here: https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html

The LINSTOR-Controller also provides a /health HTTP path that will simply return HTTP-Status 200 if the controller can access its database and all services are up and running. Otherwise it will return HTTP error status code 500 Internal Server Error.

After configuring the LINSTOR Controller to authenticate users through LDAP (or LDAPS), and the LINSTOR REST API HTTPS, you will need to enter LINSTOR commands as follows:

If you have configured LDAP authentication without also configuring LINSTOR REST API HTTPS, you will need to explicitly enable password authentication over HTTP, by using the --allow-insecure-path flag with your linstor commands. This is not recommended outside of a secured and isolated LAN, as you will be sending credentials in plain text.

The LINSTOR Controller will prompt you for the user’s password, in each of the above examples. You may optionally use the --password argument to supply the user’s password on the command line, with all the warnings of caution that would go along with doing so.

LINSTOR automatically configures quorum policies on resources when quorum is achievable. This means, whenever you have at least two diskful and one or more diskless resource assignments, or three or more diskful resource assignments, LINSTOR will enable quorum policies for your resources automatically.

Inversely, LINSTOR will automatically disable quorum policies whenever there are less than the minimum required resource assignments to achieve quorum.

This is controlled through the, DrbdOptions/auto-quorum, property which can be applied to the linstor-controller, resource-group, and resource-definition. Accepted values for the DrbdOptions/auto-quorum property are disabled, suspend-io, and io-error.

Setting the DrbdOptions/auto-quorum property to disabled will allow you to manually, or more granularly, control the quorum policies of your resources should you want to.

For example, to manually set the quorum policies of a resource-group named my_ssd_group, you would use the following commands:

You may want to disable DRBD’s quorum features completely. To do that, you would need to first disable DrbdOptions/auto-quorum on the appropriate LINSTOR object, and then set the DRBD quorum features accordingly. For example, use the following commands to disable quorum entirely on the my_ssd_group resource-group:

If a satellite is offline for a prolonged period of time, LINSTOR can be configured to declare that node as evicted. This triggers an automated reassignment of the affected DRBD resources to other nodes to ensure a minimum replica count is kept.

This feature uses the following properties to adapt the behaviour.

After the linstor-controller loses the connection to a satellite, aside from trying to reconnect, it starts a timer for that satellite. As soon as that timer exceeds DrbdOptions/AutoEvictAfterTime and all of the DRBD-connections to the DRBD-resources on that satellite are broken, the controller will check whether or not DrbdOptions/AutoEvictMaxDisconnectedNodes has been met. If it hasn’t, and DrbdOptions/AutoEvictAllowEviction is true for the node in question, the satellite will be marked as EVICTED. At the same time, the controller will check for every DRBD-resource whether the number of resources is still above DrbdOptions/AutoEvictMinReplicaCount. If it is, the resource in question will be marked as DELETED. If it isn’t, an auto-place with the settings from the corresponding resource-group will be started. Should the auto-place fail, the controller will try again later when changes that might allow a different result, such as adding a new node, have happened. Resources where an auto-place is necessary will only be marked as DELETED if the corresponding auto-place was successful.

The evicted satellite itself will not be able to reestablish connection with the controller. Even if the node is up and running, a manual reconnect will fail. It is also not possible to delete the satellite, even if it is working as it should be. The satellite can, however, be restored. This will remove the EVICTED-flag from the satellite and allow you to use it again. Previously configured network interfaces, storage pools, properties and similar entities as well as non-DRBD-related resources and resources that could not be autoplaced somewhere else will still be on the satellite. To restore a satellite, use

Should you want to instead throw everything that once was on that node, including the node itself, away, you need to use the node lost command instead.

These LINSTOR properties may be set using the set-property command and may be set on the following objects: volume, storage pool, resource, controller, or node. You can also set these QoS properties on resource groups, volume groups, resource definitions, or volume definitions. When you set a QoS property on a group or definition, resources created from the group or definition will inherit the QoS settings.

The following example shows creating a resource group, then creating a volume group, then applying QoS settings to the volume group, and then spawning resources from the resource group. A verification command will show that the spawned resources inherit the QoS settings. The example uses an assumed previously created LINSTOR storage pool named pool1. You will need to replace this name with a storage pool name that exists in your environment.

To verify that the spawned resources inherited the QoS setting, you can show the contents of the corresponding sysfs file, on a node that contributes storage to the storage pool.

A single LINSTOR volume can be composed of multiple DRBD devices. For example, DRBD with external metadata will have three backing devices: a data (storage) device, a metadata device, and the composite DRBD device (volume) provided to LINSTOR. If the data and metadata devices correspond to different backing disks, then if you set a sysfs property for such a LINSTOR volume, only the local data (storage) backing device will receive the property value in the corresponding /sys/fs/cgroup/blkio/ file. Neither the device backing DRBD’s metadata, nor the composite backing device provided to LINSTOR would receive the value. However, when DRBD’s data and its metadata share the same backing disk, QoS settings will affect the performance of both data and metadata operations.

In case a LINSTOR resource definition has an nvme-target as well as an nvme-initiator resource, both data (storage) backing devices of each node will receive the sysfs property value. In case of the target, the data backing device will be the volume of either LVM or ZFS, whereas in case of the initiator, the data backing device will be the connected nvme-device, regardless of which other LINSTOR layers, such as LUKS, NVMe, DRBD, and others (see Using LINSTOR Without DRBD), are above that.

A quick way to list available commands on the command line is to type linstor.

Further information about subcommands (e.g., list-nodes) can be retrieved in two ways:

Using the ‘help’ subcommand is especially helpful when LINSTOR is executed in interactive mode (linstor interactive).

One of the most helpful features of LINSTOR is its rich tab-completion, which can be used to complete basically every object LINSTOR knows about (e.g., node names, IP addresses, resource names, …​). In the following examples, we show some possible completions, and their results:

If tab-completion does not work out of the box, please try to source the appropriate file:

For zsh shell users, the linstor-client command can generate a zsh compilation file, that has basic support for command and argument completion.

If something goes wrong and you need help finding the cause of the issue, you can use

The command above will create a new sos-report in /var/log/linstor/controller/ on the controller node. Alternatively you can use

which will create a new sos-report and additionally downloads that report to the local machine into your current working directory.

This sos-report contains logs and useful debug-information from several sources (Linstor-logs, dmesg, versions of external tools used by LINSTOR, ip a, database dump and many more). These information are stored for each node in plain text in the resulting .tar.gz file.

For help from the community please subscribe to our mailing list located here: https://lists.linbit.com/listinfo/drbd-user

To file bug or feature request please check out our GitHub page https://github.com/linbit

Alternatively, if you need to purchase remote installation services, 24/7 support, access to certified repositories, or feature development, please contact us: +1-877-454-6248 (1-877-4LINBIT) , International: +43-1-8178292-0 | sales@linbit.com

LINBIT provides a LINSTOR Operator to commercial support customers. The Operator eases deployment of LINSTOR on Kubernetes by installing DRBD, managing Satellite and Controller pods, and other related functions.

The Operator itself is installed using a Helm v3 chart as follows:

The LINSTOR Operator can automate some basic storage set up for LINSTOR.

This section describes the different options for enabling security features available when using this operator. The following guides assume the operator is installed using Helm

The Helm charts provide a set of further customization options for advanced use cases.

You can use Prometheus to monitor LINSTOR components. The operator will set up monitoring containers along the existing components and make them available as a Service.

If you use the Prometheus Operator, the LINSTOR Operator will also set up the ServiceMonitor instances. The metrics will automatically be collected by the Prometheus instance associated to the operator, assuming watching the Piraeus namespace is enabled.

To disable exporting of metrics, set operator.satelliteSet.monitoringImage to an empty value.

The operator can configure the satellites and CSI plug-in to use an existing LINSTOR setup. This can be useful in cases where the storage infrastructure is separate from the Kubernetes cluster. Volumes can be provisioned in diskless mode on the Kubernetes nodes while the storage nodes will provide the backing disk storage.

To skip the creation of a LINSTOR Controller deployment and configure the other components to use your existing LINSTOR Controller, use the following options when running helm install:

After all pods are ready, you should see the Kubernetes cluster nodes as satellites in your LINSTOR setup.

Create a storage class referencing an existing storage pool on your storage nodes.

You can provision new volumes by creating PVCs using your storage class. The volumes will first be placed only on nodes with the given storage pool, that is, your storage infrastructure. Once you want to use the volume in a pod, LINSTOR CSI will create a diskless resource on the Kubernetes node and attach over the network to the diskful resource.

The community supported edition of the LINSTOR deployment in Kubernetes is called Piraeus. The Piraeus project provides an operator for deployment.

The following storage class contains all currently available parameters to configure the provisioned storage.

The csi.storage.k8s.io/fstype parameter sets the file system type to create for volumeMode: FileSystem PVCs. Currently supported are:

autoPlace is an integer that determines the amount of replicas a volume of this StorageClass will have. For instance, autoPlace: "3" will produce volumes with three-way replication. If neither autoPlace nor nodeList are set, volumes will be automatically placed on one node.

placementCount is an alias for autoPlace

The LINSTOR Resource Group (RG) to associate with this StorageClass. If not set, a new RG will be created for each new PVC.

storagePool is the name of the LINSTOR storage pool that will be used to provide storage to the newly-created volumes.

disklessStoragePool is an optional parameter that only effects LINSTOR volumes assigned disklessly to kubelets i.e., as clients. If you have a custom diskless storage pool defined in LINSTOR, you’ll specify that here.

A comma-separated list of layers to use for the created volumes. The available layers and their order are described towards the end of this section. Defaults to drbd,storage

Select from one of the available volume schedulers:

Control on which nodes a volume is accessible. The value for this option can take two different forms:

encryption is an optional parameter that determines whether to encrypt volumes. LINSTOR must be configured for encryption for this to work properly.

nodeList is a list of nodes for volumes to be assigned to. This will assign the volume to each node and it will be replicated among all of them. This can also be used to select a single node by hostname, but it’s more flexible to use replicasOnSame to select a single node.

clientList is a list of nodes for diskless volumes to be assigned to. Use in conjunction with nodeList.

replicasOnSame is a list of key or key=value items used as autoplacement selection labels when autoplace is used to determine where to provision storage. These labels correspond to LINSTOR node properties.

Let’s explore this behavior with examples assuming a LINSTOR cluster such that node-a is configured with the following auxiliary property zone=z1 and role=backups, while node-b is configured with only zone=z1.

If we configure a StorageClass with autoPlace: "1" and replicasOnSame: "zone=z1 role=backups", then all volumes created from that StorageClass will be provisioned on node-a, since that is the only node with all of the correct key=value pairs in the LINSTOR cluster. This is the most flexible way to select a single node for provisioning.

If we configure a StorageClass with autoPlace: "1" and replicasOnSame: "zone=z1", then volumes will be provisioned on either node-a or node-b as they both have the zone=z1 aux prop.

If we configure a StorageClass with autoPlace: "2" and replicasOnSame: "zone=z1 role=backups", then provisioning will fail, as there are not two or more nodes that have the appropriate auxiliary properties.

If we configure a StorageClass with autoPlace: "2" and replicasOnSame: "zone=z1", then volumes will be provisioned on both node-a and node-b as they both have the zone=z1 aux prop.

You can also use a property key without providing a value to ensure all replicas are placed on nodes with the same property value, with caring about the particular value. Assuming there are 4 nodes, node-a1 and node-a2 are configured with zone=a. node-b1 and node-b2 are configured with zone=b. Using autoPlace: "2" and replicasOnSame: "zone" will place on either node-a1 and node-a2 OR on node-b1 and node-b2.

replicasOnDifferent takes a list of properties to consider, same as replicasOnSame. There are two modes of using replicasOnDifferent:

Create a diskless resource on all nodes that were not assigned a diskful resource.

Do not place the resource on a node which has a resource with a name matching the regex.

fsOpts is an optional parameter that passes options to the volume’s filesystem at creation time.

mountOpts is an optional parameter that passes options to the volume’s filesystem at mount time.

Extra arguments to pass to xfs_io, which gets called before right before first use of the volume.

Parameters starting with property.linstor.csi.linbit.com/ are translated to LINSTOR properties that are set on the Resource Group associated with the StorageClass.

For example, to set DrbdOptions/auto-quorum to disabled, use:

The full list of options is available here

Advanced DRBD options to pass to LINSTOR. For example, to change the replication protocol, use DrbdOptions/Net/protocol: "A".

LINSTOR supports the volume snapshot feature, which is configured in some, but not all Kubernetes distributions.

To check if your Kubernetes distribution supports snapshots out of the box, run the following command:

In case your Kubernetes distribution does not support snapshots, you can manually add the required components from the Kubernetes Storage SIG. For convenience, you can use Helm Charts provided by the Piraeus team to add these components.

Then we can create our VolumeSnapshotClass:

Create the VolumeSnapshotClass with kubectl:

Now we will create a volume snapshot for the volume that we created above. This is done with a VolumeSnapshot:

Create the VolumeSnapshot with kubectl:

You can check that the snapshot creation was successful

Finally, we’ll create a new volume from the snapshot with a PVC.

Create the PVC with kubectl:

The following example show common scenarios where you want to optimize volume accessibility and locality. It also includes examples of how to spread volume replicas across zones in a cluster.

The LINSTOR Operator will create a backup of the database before upgrading. This makes it possible to roll back to a known state should something go wrong during the upgrade.

There are situations in which the Operator can’t create the backup automatically. This might be because:

Some versions require special steps, see below.

To create a backup of the etcd database and store it on your control host, run:

These commands will create a file save.db on the machine you are running kubectl from.

The next steps are exactly the same as in Kubernetes:

The LINSTOR container images come with the LINBIT GUI preinstalled. To expose it on your cluster, configure an OpenShift Route resource:

The GUI is now accessible at https://linstor-op-cs-storage.apps.oc.example.com/ui/

OpenShift includes a fully configured monitoring stack. While most of the monitoring stack is only intended for Openstack infrastructure, basic monitoring for LINBIT SDS is possible.

First, ensure that monitoring of user-defined projects is enabled in OpenShift by following the steps in Red Hat documentation.

Once monitoring for user-defined projects is enabled, the LINSTOR Operator automatically detects the presence of OpenShift’s Prometheus-based monitoring stack and configures the ServiceMonitor resources. The Prometheus instance will scrape metrics for DRBD and LINSTOR from all Cluster nodes.

To run LINSTOR, every Nomad agent needs to be configured to:

The LINSTOR Controller is deployed as a service with no replicas. At any point in time, there can only be one LINSTOR Controller running in a cluster. It is possible to restart the controller on a new node, provided it still has access to the database. See Creating a Highly Available LINSTOR Cluster for more information.

The following example will create a Nomad job starting a single LINSTOR Controller in datacenter dc1 and connect to an external database.

Apply the job by running:

In Nomad, the LINSTOR satellites are deployed as a system job that runs in a privileged container. In addition to the satellites, the job will also load the DRBD module along with other kernel modules used by LINSTOR.

The following example will create a Nomad job starting a LINSTOR satellite on every node in data center dc1.

Apply the job by running:

Once the linstor-controller and linstor-satellite jobs are running, you can start configuring the cluster using the linstor command line tool.

This can done:

In all cases, you need to add the satellites to your cluster and create some storage pools. For example, to add the node nomad-01.example.com and configure a LVM Thin storage pool, you would run:

To use the volume in a job, add the volume and volume_mount stanzas to the job specification:

LINSTOR can create snapshots of existing volumes, provided the underlying storage pool driver supports snapshots.

The following command creates a snapshot named snap1 of the volume vol1.

You can use a snapshot to pre-populate an existing volume with data from the snapshot

Version 5 of the plug-in drops compatibility with the legacy configuration options “storagepool” and “redundancy”. Version 5 requires a “resourcegroup” option, and obviously a LINSTOR resource group. The old options should be removed from the configuration.

Configuring LINSTOR is described in Section Configuring LINSTOR, a typical example follows:

Let’s assume the pool was set to “mypool”, and redundancy to 3.

Version 6.0.0 of the plug-in drops all code related to the redundancy setting. This is handled by LINSTOR resource groups (resourcegroup setting) for a very long time. No change should be required.

The controllervm setting, which was intended for executing a LINSTOR controller in a VM manged by LINSTOR is gone. Using drbd-reactor to realize a highly available LINSTOR controller is what we suggest.

The settings statuscache and preferlocal are now enabled by default.

With version 6 PVE added additional parameters to some functions and rightfully reset their “APIAGE”. This means that old plug-ins, while actually usable as they don’t use any of these changed functions do not work anymore. Please upgrade to plug-in version 5.2.1 at least.

To use LINSTOR in Proxmox, you will need to install the DRBD kernel module. The DRBD 9 kernel module is installed as a kernel module source package (drbd-dkms). Therefore, you will have to install the Proxmox VE kernel headers package, pve-headers, before you install the DRBD kernel module from LINBIT’s repositories. Following that order ensures that the kernel module will build properly for your kernel.

If you do not plan to install the latest Proxmox kernel, you have to install kernel headers matching your current running kernel (for example, pve-headers-$(uname -r)). If you missed this step, then you can still rebuild the drbd-dkms package against your current kernel (so long as you have installed kernel headers in advance) by entering the apt install --reinstall drbd-dkms command.

If you are a LINBIT customer, or you have an evaluation account, you can enable LINBIT’s drbd-9 repository on your node and then update your repositories by using an apt update command.

You can then install the DRBD kernel module, DRBD utilities, and the LINSTOR Proxmox plug-in by entering:

LINBIT provides a dedicated public repository for Proxmox VE users. This repository not only contains the Proxmox plug-in, but the whole DRBD SDS stack including a DRBD SDS kernel module and user-space utilities.

LINBIT’s repository can be enabled as follows, where “$PVERS” should be set to your Proxmox VE major version (for example, “7”, not “7.1”):

Modify the following sections of /etc/one/oned.conf

Add linstor to the list of drivers in the TM_MAD and DATASTORE_MAD sections:

Add new TM_MAD_CONF and DS_MAD_CONF sections:

After making these changes, restart the OpenNebula service.

The Front-End node issues commands to the Storage and Host nodes through LINSTOR.

Storage nodes hold disk images of VMs locally.

Host nodes are responsible for running instantiated VMs and typically have the storage for the images they need attached across the network through LINSTOR diskless mode.

All nodes must have DRBD9 and LINSTOR installed. This process is detailed in the User’s Guide for DRBD9

It is possible to have Front-End and Host nodes act as storage nodes in addition to their primary role provided that they the meet all the requirements for both roles.

The oneadmin, “Cloud Administrator”, user account must have passwordless sudo access to the mkfs command on the Storage nodes

Create a datastore configuration file named ds.conf and use the onedatastore tool to create a new datastore based on that configuration. There are two mutually exclusive deployment options: LINSTOR_AUTO_PLACE and LINSTOR_DEPLOYMENT_NODES. If both are configured, LINSTOR_AUTO_PLACE is ignored. For both of these options, BRIDGE_LIST must be a space separated list of all storage nodes in the LINSTOR cluster.

Since version 1.0.0 LINSTOR supports resource groups. A resource group is a centralized point for settings that all resources linked to that resource group share.

Create a resource group and volume group for your datastore, it is mandatory to specify a storage-pool within the resource group, otherwise monitoring space for OpenNebula will not work. Here we create one with 2 node redundancy and use a created opennebula-storagepool:

Now add a OpenNebula datastore using the LINSTOR plug-in:

The following attributes are deprecated and were removed in version 2.0.

LINSTOR driver can also be used as a system datastore, configuration is pretty similar to normal datastores, with a few changes:

Also add the new sys datastore id to the COMPATIBLE_SYS_DS to your image datastores (COMMA separated), otherwise the scheduler will ignore them.

If you want live migration with volatile disks you need to enable the --unsafe option for KVM, see: opennebula-doc

If this is a fresh installation, skip this section and continue with Installing the LINSTOR Driver.

Starting with OpenStack Stein, the LINSTOR driver is part of the Cinder project. While the driver can be used as is, it might be missing features or fixes available in newer version. Due to OpenStacks update policy for stable versions, most improvements to the driver will not get back-ported to older stable releases.

LINBIT maintains a fork of the Cinder repository with all improvements to the LINSTOR driver backported to the supported stable versions. Currently, these are:

The exact steps to enable the LINSTOR Driver depend on your OpenStack distribution. In general, the python-linstor package needs to be installed on all hosts running the Cinder volume service. The next section will cover the installation process for common OpenStack distributions.

The Transport Protocol in Cinder is how clients (for example nova-compute) access the actual volumes. With LINSTOR, you can choose between two different drivers that use different transports.

To verify that all back ends are up and running, you can use the OpenStack command line client:

If you have the Horizon GUI deployed, check Admin > System Information > Block Storage Service instead.

In the above example all configured services are enabled and up. If there are any issues, please check the logs of the Cinder Volume service.

Each volume type can be customized by adding properties or “Extra Specs” as they are called in the Horizon GUI.

To add a property to a volume type on the command line use:

Alternatively, you can set the property using the GUI by navigating tp Admin > Volume > Volume Types. In the Actions column, open the dropdown menu and click the View Extra Specs button. This opens a dialog you can use to create, edit and delete properties.

Every volume and snapshot has a Messages tab in the Horizon dashboard. In case of errors, the list of messages can be used as a starting point for further investigation. Some common messages in case of errors:

There was an error creating a new volume. Check the Cinder Volume service logs for more details

If this is the only error message, this means the Cinder Scheduler could not find a volume back end suitable for creating the volume. This is most likely because:

The LINSTOR driver is called as part of the Cinder Volume service.

Some issues will not be logged in the Cinder Service but in the actual consumer of the volumes, most likely the compute service (Nova). As with the volume service, the exact host and location to check depends on your OpenStack distribution:

On node alpha:

On node bravo:

The LINSTOR cluster needs to be set up as a prerequisite to using LINSTOR Gateway. For more detail regarding LINSTOR’s configuration options, please refer to the LINSTOR User’s Guide.

For both iSCSI and NFS, a LINSTOR storage-pool , resource-group and volume-group for LINSTOR Gateway needs to be created before use. The following section provides example commands for setting up the prerequisites in a three-node LINSTOR cluster.

Create a LVM backed storage-pool on each node using the physical device /dev/sdb:

Create resource-groups and volume-groups backed by the storage-pool created in the previous command:

LINSTOR Gateway requires modification of the LINSTOR Satellite’s configuration on each Satellite node. Edit or create the following file using your preferred text editor:

Add the following content to the Satellite configuration:

Save the changes to the Satellite configuration and restart the satellite service on all nodes to load the changes.

DRBD Reactor is a daemon that will orchestrate the iSCSI, NFS or NVMe-oF resources in our cluster. It must be installed and the service must be enabled on all nodes in the cluster.

DRBD Reactor’s main function could be summarized as such: each DRBD Reactor daemon tries to take over DRBD Reactor managed services. The daemon that wins the race prevents other nodes from activating the services until the original winner is no longer able to. When that happens, a new winner will take over DRBD Reactor managed services, therefore achieving high-availability.

For details regarding DRBD Reactor’s installation and configuration options, refer to the DRBD Reactor GitHub page.

Start and enable the DRBD Reactor service on all nodes:

In addition to this, DRBD Reactor needs to be configured to automatically reload when its configuration changes. Run this snippet on all nodes, as described in the DRBD Reactor README:

DRBD Reactor uses Pacemaker’s resource agents when integrated with LINSTOR Gateway. You also need to install resource agents on all nodes in the cluster. This package is named resource-agents in both RPM and DEB based distributions:

LINSTOR Gateway requires the LIO iSCSI implementation be installed on all nodes.

The targetcli package can be installed on RPM based systems using the following command:

For NFS support in LINSTOR Gateway, nfs-server utilities need to be installed on each node.

The nfs-utils package can be installed on RPM based systems using the following command:

The NFS server should not be enabled in systemd since that will conflict with DRBD Reactors ability to manage the service. Disable the nfs-server service and check that it has been disabled using the following commands:

Ensure that the output of the command above lists the service as inactive and disabled:

Let’s check that all the required components are present. This guide assumes you already installed and configured a LINSTOR cluster complete with storage-pools, resource-groups, and volume-groups before using linstor-gateway.

In addition to the initialized LINSTOR cluster, the following tools need to be present on all nodes:

LINSTOR Gateway provides a utility to check that the prerequisite tools are present:

This command should print something similar to the output below if you installed all of the required components. If an error is reported, you must resolve the error before proceeding.

Verify that the LINSTOR cluster is initialized properly by comparing your outputs are similar to the outputs in the commands below.

Ensure all your LINSTOR nodes are listed as a Satellite or Combined type, and that you have 3 (or more) to support quorum: