LINSTOR was originally developed to manage DRBD resources. While you can still use LINSTOR to make managing DRBD more convenient, LINSTOR has evolved and it is often integrated with software stacks higher up to provide persistent storage more easily and more flexibly than would otherwise be possible within those stacks.

LINSTOR can be used on its own or you can integrate it with other platforms, such as Kubernetes, OpenShift, OpenNebula, OpenStack, Proxmox VE, and others. LINSTOR runs on bare-metal on-premise hardware, or you can use it within virtual machines (VMs), containers, clouds, or hybrid environments.

LINSTOR can work with the following storage provider and related technologies:

By using LINSTOR, you can work with these technologies on their own or else in various meaningful combinations.

The linstor-controller service relies on a database that holds all the configuration information for the whole cluster. A node or container running the LINSTOR controller software is responsible for resource placement, resource configuration, and orchestration of any operational processes that require a view of the whole cluster.

Multiple controllers can be used for LINSTOR, for example, when setting up a highly available LINSTOR cluster, but only one controller can be active.

As mentioned earlier, the LINSTOR controller operates on a separate plane from the data plane that it manages. You can stop the controller service, update or reboot the controller node, and still have access to your data hosted on the LINSTOR satellite nodes. While you can still access and serve the data on your LINSTOR satellite nodes, without a running controller node, you will not be able to perform any LINSTOR status or management tasks on the satellite nodes.

The linstor-satellite service runs on each node where LINSTOR consumes local storage or provides storage to services. It is stateless and receives all the information it needs from the node or container running the LINSTOR controller service. The LINSTOR satellite service runs programs like lvcreate and drbdadm. It acts like an agent on a node or in a container that carries out instructions that it receives from the LINSTOR controller node or container.

When you need to interface with LINSTOR, you can send instructions to the active LINSTOR controller by using one of its user interfaces (UIs): the LINSTOR client, or the LINBIT SDS GUI.

Both of these UIs rely on the LINSTOR REST API.

LINSTOR takes an object-oriented approach to software-defined storage (SDS). LINSTOR objects are the end result that LINSTOR presents to the user or application to consume or build upon.

The most commonly used LINSTOR objects are explained below and a full list of objects follows.

While definitions and groups are also LINSTOR objects, they are a special kind. Definition and group objects can be thought of as profiles or templates. These template objects are used to create child objects that will inherit their parent object’s attributes. They might also have attributes that can affect child objects but are not attributes of the child objects themselves. These attributes could be things such as the TCP port to use for DRBD replication or the volume number that a DRBD resource should use.

The following are some general rules for object hierarchy in LINSTOR:

This section uses diagrams to represent the hierarchical relationships between some of the most frequently used LINSTOR objects. Because of the number of LINSTOR objects and their interconnectedness, multiple diagrams are shown first rather than a single diagram, to simplify the conceptualization.

The next diagram shows the relationships between LINSTOR group objects on a single satellite node.

While the two preceding diagrams show higher-lower relationships between common LINSTOR objects, you can also think of some LINSTOR objects as having parent-child relationships. The next diagram introduces this kind of relationship between LINSTOR objects by using a storage pool definition (parent object) and a storage pool (child object) as an example. A parent object can have multiple child objects, as shown in the following diagram.

Having introduced the concept of parent-child relationships in a conceptual diagram, the next diagram is a modified version of the second diagram with some of those relationships added for groups and definitions. This modified diagram also incorporates some of the higher-lower relationships that were shown in the first diagram.

The next diagram synthesizes the relationship concepts of the preceding diagrams while also introducing new LINSTOR objects related to snapshots and connections. With the many objects and criss-crossing lines, the reason for building up to this diagram should be apparent.

Even with its seeming complexity, the preceding diagram is still a simplification and not an all-encompassing representation of the possible relationships between LINSTOR objects. As listed earlier, there are more LINSTOR objects than are shown in the diagram^[3].

The good news is that you do not need to memorize the preceding diagram to work with LINSTOR. It could be useful to refer to though if you are trying to troubleshoot attributes that you have set on LINSTOR objects and their inheritance and effects on other LINSTOR objects in your cluster.

LINSTOR is packaged in both the RPM and the DEB variants:

For further details about these packages see the Installable Components section above.

You can configure LINSTOR to encrypt storage volumes, by using LUKS (dm-crypt), as detailed in the Encrypted Volumes section of this user’s guide. Refer to the LUKS and dm-crypt projects for FIPS compliance status.

You can also configure LINSTOR to encrypt communication traffic between a LINSTOR satellite and a LINSTOR controller, by using SSL/TLS, as detailed in the Secure Satellite Connections section of this user’s guide.

LINSTOR can also interface with Self-Encrypting Drives (SEDs) and you can use LINSTOR to initialize an SED drive. LINSTOR stores the drive’s password as a property that applies to the storage pool associated with the drive. LINSTOR encrypts the SED drive password by using the LINSTOR master passphrase that you must create first.

By default, LINSTOR uses the following cryptographic algorithms:

A FIPS compliant version of LINSTOR is available for the use cases mentioned in this section. If you or your organization require FIPS compliance at this level, please contact [email protected] for details.

To use LINSTOR to create storage volumes, you will need to install a volume manager, either LVM or ZFS, if one is not already installed on your system.

If you are a LINBIT customer, you can download a LINBIT created helper script and run it on your nodes to:

Enabling LINBIT package repositories will give you access to LINBIT software packages, DRBD kernel modules, and other related software such as cluster managers and OCF scripts. You can then use a package manager to fetch, install, and manage updating installed packages.

After registering your node and enabling the drbd-9 LINBIT package repository, you can use a DEB, RPM, or YaST2 based package manager to install LINSTOR and related components.

The LINSTOR project’s GitHub page is here: https://github.com/LINBIT/linstor-server.

LINBIT also has downloadable archived files of source code for LINSTOR, DRBD, and more, available here: https://linbit.com/linbit-software-download-page-for-linstor-and-drbd-linux-driver/.

On each host contributing storage, you need to create either an LVM volume group (VG) or a ZFS zPool. The VGs and zPools identified with one LINSTOR storage pool name may have different VG or zPool names on the hosts, but do yourself a favor, for coherency, use the same VG or zPool name on all nodes.

After creating a volume group on each of your nodes, you can create a storage pool that is backed by the volume group on each of your nodes, by entering the following commands:

To list your storage pools you can enter:

or using LINSTOR abbreviated notation:

In clusters where you have only one kind of storage and the capability to hot swap storage devices, you may choose a model where you create one storage pool per physical backing device. The advantage of this model is to confine failure domains to a single storage device.

Both the Exos and LVM2 storage providers offer the option of multiple server nodes directly connected to the storage array and drives. With LVM2 the external locking service (lvmlockd) manages volume groups created with the –shared options with vgcreate. The --shared-space can be used when configuring a LINSTOR pool to use the same LVM2 volume group accessible by two or more nodes. The example below shows using the LVM2 volume group UUID as the shared space identifier for a pool accessible by nodes alpha and bravo:

Exos pools will use the Exos pool serial number by default for the shared-space identifier.

Since linstor-server 1.5.2 and a recent linstor-client, LINSTOR can create LVM/ZFS pools on a satellite for you. The linstor-client has the following commands to list possible disks and create storage pools, but such LVM/ZFS pools are not managed by LINSTOR and there is no delete command, so such action must be done manually on the nodes.

Will give you a list of available disks grouped by size and rotational(SSD/Magnetic Disk).

It will only show disks that pass the following filters:

With the create-device-pool command you can create a LVM pool on a disk and also directly add it as a storage pool in LINSTOR.

If the --storage-pool option was provided, LINSTOR will create a storage pool with the given name.

For more options and exact command usage please check the linstor-client help.

LINSTOR has the following supported storage plug-ins at the time of writing:

With the resource create command you may assign a resource definition to named nodes explicitly.

It is possible to have LINSTOR select nodes and storage pools to deploy a resource to, by using the linstor resource create --auto-place or linstor resource-definition auto-place commmands. This section will use the resource create --auto-place command in examples. However, you can use either command to produce the same results.

In the following example, the value after the --auto-place option tells LINSTOR how many replicas you want to have. The --storage-pool option should be obvious.

Maybe not so obvious is that you may omit the --storage-pool option, then LINSTOR may select a storage pool on its own. The selection follows these rules:

The remaining storage pools will be rated by different strategies. LINSTOR has currently three strategies:

The scores of the strategies will be normalized, weighted and summed up, where the scores of minimizing strategies will be converted first to allow an overall maximization of the resulting score.

The weights of the strategies can be configured with the following command:

The strategy names are listed above and the weight can be an arbitrary decimal.

Finally, LINSTOR tries to find the best matching group of storage pools meeting all requirements. This step also considers other auto-placement restrictions such as --replicas-on-same, --replicas-on-different, --do-not-place-with, --do-not-place-with-regex, --layer-list, and --providers.

You can delete a resource definition by using the command:

This will remove the named resource definition from the entire LINSTOR cluster. The resource is removed from all nodes and the resource entry is marked for removal from LINSTOR’s database tables. After LINSTOR has removed the resource from all the nodes, the resource entry is removed from LINSTOR’s database tables.

You can delete a resource using the command:

Unlike deleting a resource definition, this command will only delete a LINSTOR resource from the node (or nodes) that you specify. The resource is removed from the node and the resource entry is marked for removal from LINSTOR’s database tables. After LINSTOR has removed the resource from the node, the resource entry is removed from LINSTOR’s database tables.

Deleting a LINSTOR resource may have implications for a cluster, beyond just removing the resource. For example, if the resource is backed by a DRBD layer, removing a resource from one node in a three node cluster could also remove certain quorum related DRBD options, if any existed for the resource. After removing such a resource from a node in a three node cluster, the resource would no longer have quorum as it would now only be deployed on two nodes in the three node cluster.

After running a linstor resource delete command to remove a resource from a single node, you might see informational messages such as:

Also unlike deleting a resource definition, you can delete a resource while there are existing snapshots of the resource’s storage pool. Any existing snapshots for the resource’s storage pool will persist.

You can delete a resource group by using the command:

As you might expect, this command deletes the named resource group. You can only delete a resource group if it has no associated resource definitions, otherwise LINSTOR will present an error message, such as:

To resolve this error so that you can delete the resource group, you can either delete the associated resource definitions, or your can move the resource definitions to another (existing) resource group:

You can find which resource definitions are associated with your resource group by entering the following command:

To backup the LINSTOR database to a new file named db_export in your home directory, enter the following commands:

After backing up the database, you can copy the backup file to a safe place.

The resulting database backup is a plain JSON document, containing not just the actual data, but also some metadata about when the backup was created, from which database, and other information.

Restoring the database from a previously made backup is similar to export-db from the previous section.

For example, to restore the previously made backup from the db_export file, enter the following commands:

You can only import a database from a previous backup if the currently installed version of LINSTOR is the same (or higher) version as the version that you created the backup from. If the currently installed LINSTOR version is higher than the version that the database backup was created from, when you import the backup the data will be restored with the same database scheme of the version used during the export. Then, the next time that the controller starts, the controller will detect that the database has an old scheme and it will automatically migrate the data to the scheme of the current version.

Since the exported database file contains some metadata, an exported database file can be imported into a different database type than it was exported from.

This allows the user to convert, for example, from an etcd setup to an SQL based setup. There is no special command for converting the database format. You only have to specify the correct linstor.toml configuration file by using the --config-directory argument (or updating the default /etc/linstor/linstor.toml and specifying the database type that you want to use before importing). See the LINSTOR User’s Guide for more information about specifying a database type. Regardless of the type of database that the backup was created from, it will be imported in the database type that is specified in the linstor.toml configuration file.

For configuring the highly available (HA) storage, you can use LINSTOR itself. One of the benefits of having the storage under LINSTOR control is that you can easily extend the HA storage to new cluster nodes.

First, create a new resource that is 200MiB in size and set the necessary DRBD options, as shown in the example commands below. You will need to adapt the storage pool name to match an existing storage pool in your environment.

It is crucial that your cluster qualifies for auto-quorum and uses the io-error policy (see Section Auto-Quorum Policies), and that auto-promote is disabled.

From now on it is assumed that the resource’s name is linstor_db. After creating the linstor_db resource, you can move the LINSTOR database to the new storage and create a systemd mount service. First, stop the current controller service and disable it, as it will be managed by DRBD Reactor later.

Next, create the systemd mount service.

Copy the /etc/systemd/system/var-lib-linstor.mount mount file to all the cluster nodes that you want to have the potential to run the LINSTOR controller service (standby controller nodes). Again, do not systemctl enable any of these services because DRBD Reactor will manage them.

The next step is to install LINSTOR controllers on all nodes that have access to the linstor_db DRBD resource (as they need to mount the DRBD volume) and which you want to become a possible LINSTOR controller. It is important that the controllers are manged by drbd-reactor, so verify that the linstor-controller.service is disabled on all nodes! To be sure, execute systemctl disable linstor-controller on all cluster nodes and systemctl stop linstor-controller on all nodes except the one it is currently running from the previous step. Also verify that you have set chattr +i /var/lib/linstor on all potential controller nodes if you use LINSTOR version equal or greater to 1.14.0.

For starting and stopping the mount service and the linstor-controller service we use drbd-reactor. Install this component on all nodes that could become a LINSTOR controller and edit their /etc/drbd-reactor.d/linstor_db.toml configuration file. It should contain an enabled promoter plug-in section like this:

Depending on your requirements you might also want to set an on-stop-failure action and set stop-services-on-exit.

After that restart drbd-reactor and enable it on all the nodes you configured it.

Check that there are no warnings from drbd-reactor service in the logs by running systemctl status drbd-reactor. As there is already an active LINSTOR controller things will just stay the way they are. Run drbd-reactorctl status linstor_db to check the health of the linstor_db target unit.

The last but nevertheless important step is to configure the LINSTOR satellite services to not delete (and then regenerate) the resource file for the LINSTOR controller DB at its startup. Do not edit the service files directly, but use systemctl edit. Edit the service file on all nodes that could become a LINSTOR controller and that are also LINSTOR satellites.

After this change you should execute systemctl restart linstor-satellite on all satellite nodes.

NVMe-oF/NVMe-TCP allows LINSTOR to connect diskless resources to a node with the same resource where the data is stored over NVMe fabrics. This leads to the advantage that resources can be mounted without using local storage by accessing the data over the network. LINSTOR is not using DRBD in this case, and therefore NVMe resources provisioned by LINSTOR are not replicated, the data is stored on one node.

To use NVMe-oF/NVMe-TCP with LINSTOR the package nvme-cli needs to be installed on every node which acts as a satellite and will use NVMe-oF/NVMe-TCP for a resource. For example, on a DEB-based system, to install the package, enter the following command:

To make a resource which uses NVMe-oF/NVMe-TCP an additional parameter has to be given as you create the resource-definition:

To use NVMe-TCP rather than the default NVMe-oF, the following property needs to be set:

The property NVMe/TRType can alternatively be set on resource-group or controller level.

Next, create the volume-definition for our resource:

Before you create the resource on your nodes you have to know where the data will be stored locally and which node accesses it over the network.

First we create the resource on the node where our data will be stored:

On the nodes where the resource-data will be accessed over the network, the resource has to be defined as diskless:

Now you can mount the resource nvmedata on one of your nodes.

Since version 1.5.0 the additional Layer openflex can be used in LINSTOR. From LINSTOR’s perspective, the OpenFlex Composable Infrastructure takes the role of a combined layer acting as a storage layer (like LVM) and also providing the allocated space as an NVMe target. OpenFlex has a REST API which is also used by LINSTOR to operate with.

As OpenFlex combines concepts of LINSTOR’s storage as well as NVMe-layer, LINSTOR was added both, a new storage driver for the storage pools as well as a dedicated openflex layer which uses the mentioned REST API.

In order for LINSTOR to communicate with the OpenFlex-API, LINSTOR needs some additional properties, which can be set once on controller level to take LINSTOR-cluster wide effect:

Once that is configured, we can now create LINSTOR objects to represent the OpenFlex architecture. The theoretical mapping of LINSTOR objects to OpenFlex objects are as follows: Obviously an OpenFlex storage pool is represented by a LINSTOR storage pool. As the next thing above a LINSTOR storage pool is already the node, a LINSTOR node represents an OpenFlex storage device. The OpenFlex objects above storage device are not mapped by LINSTOR.

When using NVMe, LINSTOR was designed to run on both sides, the NVMe target as well as on the NVMe initiator side. In the case of OpenFlex, LINSTOR cannot (or even should not) run on the NVMe target side as that is completely managed by OpenFlex. As LINSTOR still needs nodes and storage pools to represent the OpenFlex counterparts, the LINSTOR client was extended with special node create commands since 1.0.14. These commands not only accept additionally needed configuration data, but also starts a “special satellite” besides the already running controller instance. These special satellites are completely LINSTOR managed. They will shut down when the controller shuts down and will be started again when the controller starts.

The new client command for creating a “special satellite” representing an OpenFlex storage device is:

The arguments are as follows:

The last step of the configuration is the creation of LINSTOR storage pools:

Once all necessary storage pools are created in LINSTOR, the next steps are similar to the usage of using an NVMe resource with LINSTOR. Here is a complete example:

A DM-Writecache device is composed of two devices: one storage device and one cache device. LINSTOR can setup such a writecache device, but needs some additional information, like the storage pool and the size of the cache device.

The two properties set in the examples are mandatory, but can also be set on controller level which would act as a default for all resources with WRITECACHE in their --layer-list. However, please note that the Writecache/PoolName refers to the corresponding node. If the node does not have a storage pool named pmempool you will get an error message.

The 4 mandatory parameters required by DM-Writecache are either configured through a property or figured out by LINSTOR. The optional properties listed in the mentioned link can also be set through a property. Please see linstor controller set-property --help for a list of Writecache/* property-keys.

Using --layer-list DRBD,WRITECACHE,STORAGE while having DRBD configured to use external metadata, only the backing device will use a writecache, not the device holding the external metadata.

LINSTOR can also setup a DM-Cache device, which is very similar to the DM-Writecache from the previous section. The major difference is that a cache device is composed by three devices: one storage device, one cache device and one meta device. The LINSTOR properties are quite similar to those of the writecache but are located in the Cache namespace:

Please see linstor controller set-property --help for a list of Cache/* property-keys and default values for omitted properties.

Using --layer-list DRBD,CACHE,STORAGE while having DRBD configured to use external metadata, only the backing device will use a cache, not the device holding the external metadata.

The storage layer will provide new devices from well known volume managers like LVM, ZFS or others. Every layer combination needs to be based on a storage layer, even if the resource should be diskless – for that type there is a dedicated diskless provider type.

For a list of providers with their properties please see Storage Providers.

For some storage providers LINSTOR has special properties:

A storage pool with the type remote SPDK can only be created on a “special satellite”. For this you first need to start a new satellite using the command:

This will start a new satellite instance running on the same machine as the controller. This special satellite will do all the REST based RPC communication towards the remote SPDK proxy. As the help message of the LINSTOR command shows, the administrator might want to use additional settings when creating this special satellite:

The difference between the --api-* and their corresponding --api-\*-env versions is that the version with the -env ending will look for an environment variable containing the actual value to use whereas the --api-\* version directly take the value which is stored in the LINSTOR property. Administrators might not want to save the --api-pw in plain text, which would be clearly visible using commands like linstor node list-property <nodeName>.

Once that special satellite is up and running the actual storage pool can be created:

Whereas node_name is self-explanatory, name is the name of the LINSTOR storage pool and driver_pool_name refers to the SPDK logical volume store.

Once this remotespdk storage pool is created the remaining procedure is quite similar as using NVMe: First the target has to be created by creating a simple “diskful” resource followed by a second resource having the --nvme-initiator option enabled.

To use multiple network paths for DRBD setups, the PrefNic property is not sufficient. Instead the linstor node interface and linstor resource-connection path commands should be used, as shown below.

The first four commands in the example define a network interface (nic1 and nic2) for each node (alpha and bravo) by specifying the network interface’s IP address. The last two commands create network path entries in the DRBD .res file that LINSTOR generates. This is the relevant part of the resulting .res file:

Below are details about the commands.

Before LINSTOR can encrypt any volume a master passphrase needs to be created. This can be done with the linstor-client.

crypt-create-passphrase will wait for the user to input the initial master passphrase (as all other crypt commands will with no arguments).

If you ever want to change the master passphrase this can be done with:

The luks layer can be added when creating the resource-definition or the resource itself, whereas the former method is recommended since it will be automatically applied to all resource created from that resource-definition.

To enter the master passphrase (after controller restart) use the following command:

It is possible to automate the process of creating and re-entering the master passphrase.

To use this, either an environment variable called MASTER_PASSPHRASE or an entry in /etc/linstor/linstor.toml containing the master passphrase has to be created.

The required linstor.toml looks like this:

If either one of these is set, then every time the controller starts it will check whether a master passphrase already exists. If there is none, it will create a new master passphrase as specified. Otherwise, the controller enters the passphrase.

Some evacuation cases may need special planning. For example, if you are evacuating more than one node, you can exclude the nodes from participating in LINSTOR’s resource autoplacer. You can do this by using the following command on each node that you want to evacuate:

This ensures that LINSTOR will not place resources from a node that you are evacuating onto another node that you plan on evacuating.

If you already ran a node evacuate command that has either completed or still has resources in an “Evacuating” state, you can remove the “Evacuating” state from a node by using the node restore command. This will work so long as you have not yet run a node delete command.

After restoring the node, you should use the node set-property <node_name> AutoplaceTarget true command, if you previously set the AutoplaceTarget property to “false”. This way, LINSTOR can again place resources onto the node automatically, to fulfill placement count properties that you might have set for resources in your cluster.

Assuming a resource definition named ‘resource1’ which has been placed on some nodes, a snapshot can be created as follows:

This will create snapshots on all nodes where the resource is present. LINSTOR will ensure that consistent snapshots are taken even when the resource is in active use.

Setting the resource-definition property AutoSnapshot/RunEvery LINSTOR will automatically create snapshots every X minute. The optional property AutoSnapshot/Keep can be used to clean-up old snapshots which were created automatically. No manually created snapshot will be cleaned-up / deleted. If AutoSnapshot/Keep is omitted (or ⇐ 0), LINSTOR will keep the last 10 snapshots by default.

The following steps restore a snapshot to a new resource. This is possible even when the original resource has been removed from the nodes where the snapshots were taken.

First define the new resource with volumes matching those from the snapshot:

At this point, additional configuration can be applied if necessary. Then, when ready, create resources based on the snapshots:

This will place the new resource on all nodes where the snapshot is present. The nodes on which to place the resource can also be selected explicitly; see the help (linstor snapshot resource restore -h).

LINSTOR can roll a resource back to a snapshot state. The resource must not be in use. That is, it may not be mounted on any nodes. If the resource is in use, consider whether you can achieve your goal by restoring the snapshot instead.

Rollback is performed as follows:

A resource can only be rolled back to the most recent snapshot. To roll back to an older snapshot, first delete the intermediate snapshots.

An existing snapshot can be removed as follows:

Snapshots can be shipped between LINSTOR nodes or between different LINSTOR clusters, as well as to an S3 storage such as Amazon S3 or min.io.

The following tools need to be installed on the satellites that are going to send or receive snapshots:

You create a backup shipping schedule by using the LINSTOR client schedule create command and defining the frequency of backup shipping using cron syntax. You also need to set options that name the schedule and define various aspects of the backup shipping, such as on-failure actions, the number of local and remote backup copies to keep, and whether to also schedule incremental backup shipping.

At a minimum, the command needs a schedule name and a full backup cron schema to create a backup shipping schedule. An example command would look like this:

You can modify a backup shipping schedule by using the LINSTOR client schedule modify command. The syntax for the command is the same as that for the schedule create command. The name that you specify with the schedule modify command must be an already existing backup schedule. Any options to the command that you do not specify will retain their existing values. If you want to set the keep-local or keep-remote options back to their default values, you can set them to “all”. If you want to set the max-retries option to its default value, you can set it to “forever”.

Your physical storage is not infinite and your remote storage has a cost, so you will likely want to set limits on the number of snapshots and backups you keep.

Both the --keep-remote and --keep-local options deserve special mention as they have implications beyond what may be obvious. Using these options, you specify how many snapshots or full backups should be kept, either on the local source or the remote destination.

You can list your backup shipping schedules by using the linstor schedule list command.

For example:

The LINSTOR client schedule delete command completely deletes a backup shipping schedule LINSTOR object. The command’s only argument is the schedule name that you want to delete. If the deleted schedule is currently creating or shipping a backup, the scheduled shipping process is stopped. Depending on at which point the process stops, a snapshot, or a backup, or both, might not be created and shipped.

This command does not affect previously created snapshots or successfully shipped backups. These will be retained until they are manually deleted.

You can use the LINSTOR client backup schedule enable command to enable a previously created backup shipping schedule. The command has the following syntax:

To disable a previously enabled backup shipping schedule, you use the LINSTOR client backup schedule disable command. The command has the following syntax:

If you include the option specifying either a resource group or resource definition, as described in the backup schedule enable command example above, then you disable the schedule only for that resource group or resource definition.

For example, if you omitted specifying a resource group or resource definition in an earlier backup schedule enable command, LINSTOR would schedule backup shipping for all its deployed resources that can make snapshots. Your disable command would then only affect the resource group or resource definition that you specify with the command. The backup shipping schedule would still apply to any deployed LINSTOR resources besides the specified resource group or resource definition.

The same as for the backup schedule enable command, if you specify neither a resource group nor a resource definition, then LINSTOR disables the backup shipping schedule at the controller level for all deployed LINSTOR resources.

You can use the linstor backup schedule delete command to granularly delete either a specified resource definition or a resource group from a backup shipping schedule, without deleting the schedule itself. This command has the same syntax and arguments as the backup schedule disable command. If you specify neither a resource group nor a resource definition, the backup shipping schedule you specify will be deleted at the controller level.

It may be helpful to think about the backup schedule delete command as a way that you can remove a backup shipping schedule-remote pair from a specified LINSTOR object level, either a resource definition, a resource group, or at the controller level if neither is specified.

The backup schedule delete command does not affect previously created snapshots or successfully shipped backups. These will be retained until they are manually deleted, or until they are removed by the effects of a still applicable keep-local or keep-remote option.

You might want to use this command when you have disabled a backup schedule for multiple LINSTOR object levels and later want to affect a granular change, where a backup schedule enable command might have unintended consequences.

For example, consider a scenario where you have a backup schedule-remote pair that you enabled at a controller level. This controller has a resource group, myresgroup that has several resource definitions, resdef1 through resdef9, under it. For maintenance reasons perhaps, you disable the schedule for two resource definitions, resdef1 and resdef2. You then realize that further maintenance requires that you disable the backup shipping schedule at the resource group level, for your myresgroup resource group.

After completing some maintenance, you are able to enable the backup shipping schedule for resdef3 through resdef9, but you are not yet ready to resume (enable) backup shipping for resdef1 and resdef2. You can enable backup shipping for each resource definition individually, resdef3 through resdef9, or you can use the backup schedule delete command to delete the backup shipping schedule from the resource group, myresgroup. If you use the backup schedule delete command, backups of resdef3 through resdef9 will ship again because the backup shipping schedule is enabled at the controller level, but resdef1 and resdef2 will not ship because the backup shipping schedule is still disabled for them at the resource definition level.

When you complete your maintenance and are again ready to ship backups for resdef1 and resdef2, you can delete the backup shipping schedule for those two resource definitions to return to your starting state: backup shipping scheduled for all LINSTOR deployed resources at the controller level. To visualize this it may be helpful to refer to the decision tree diagram for how LINSTOR decides whether or not to ship a backup in the How the LINSTOR Controller Determines Scheduled Backup Shipping subsection.

You can list backup schedules by resource, using the LINSTOR client schedule list-by-resource command. This command will show LINSTOR resources and how any backup shipping schedules apply and to which remotes they are being shipped. If resources are not being shipped then the command will show:

If resources have schedule-remote-pairs and are being shipped, the command output will show when the last backup was shipped and when the next backup is scheduled to ship. It will also show whether the next and last backup shipments were full or incremental backups. Finally, the command will show when the next planned incremental (if any) and full backup shipping will occur.

You can use the --active-only flag with the schedule list-by-resource command to filter out all resources that are not being shipped.

To determine if the LINSTOR Controller will ship a deployed LINSTOR resource with a certain backup schedule for a given remote destination, the LINSTOR Controller uses the following logic:

As the diagram shows, enabled or disabled backup shipping schedules have effect in the following order:

A backup shipping schedule-remote pair that is enabled or disabled at a preceding level will override the enabled or disabled status for the same schedule-remote pair at a later level.

To determine how a LINSTOR resource will be affected by scheduled backup shipping, you can use the LINSTOR client schedule list-by-resource-details command for a specified LINSTOR resource.

The command will output a table that shows on what LINSTOR object level a backup shipping schedule is either not set (empty cell), enabled, or disabled.

By using this command, you can determine on which level you need to make a change to enable, disable, or delete scheduled backup shipping for a resource.

Example output could look like this:

The LINSTOR resource object is an exception to the general syntax for setting DRBD options for LINSTOR objects. With the LINSTOR resource object, you can use the drbd-peer-options to set DRBD options at the connection level between the two nodes that you specify. Specifying drbd-peer-options for a LINSTOR resource object between two nodes is equivalent to using the`linstor resource-connection drbd-peer-options` for a resource between two nodes.

For example, to set the DRBD maximum buffer size to 8192 at a connection level, for a resource named backups, between two nodes, node-0 and node-1, enter:

The command above is equivalent to the following:

Indeed, when using the linstor --curl command to examine the two commands actions on the LINSTOR REST API, the output is identical:

The connection section of the LINSTOR-generated resource file backups.res on node-0 will look something like this:

You can use the drbd-peer-options argument to set DRBD options at a connection level, between two nodes, for example:

The preceding command would set the DRBD ping-timeout option to 29.9 seconds at a connection level between two nodes, node-0 and node-1.

You can verify a LINSTOR object’s set properties by using the list-properties command, for example:

To remove a previously set DRBD option, prefix the option name with unset-. For example:

The same syntax applies to any drbd-peer-options set either on a LINSTOR resource, resource connection, or node connection. For example:

Removing a DRBD option or DRBD peer option will return the option to its default value. Refer to the linstor <LINSTOR_object> drbd-options --help (or drbd-peer-options --help) command output for the default values of options. You can also refer to the drbd.conf-9.0 man page to get information about DRBD options.

To move a resource between nodes without reducing redundancy at any point, LINSTOR’s disk migrate feature can be used. First create a diskless resource on the target node, and then add a disk using the --migrate-from option. This will wait until the data has been synced to the new disk and then remove the source disk.

For example, to migrate a resource backups from ‘alpha’ to ‘bravo’:

LINSTOR can also be configured to automatically enable the above mentioned Proxy connection between two nodes. For this automation, LINSTOR first needs to know on which site each node is.

As the Site property might also be used for other site-based decisions in future features, the DrbdProxy/AutoEnable also has to be set to true:

This property can also be set on node, resource-definition, resource and resource-connection level (from left to right in increasing priority, whereas the controller is the left-most, that is, the least prioritized level).

Once this initialization steps are completed, every newly created resource will automatically check if it has to enable DRBD proxy to any of its peer-resources.

A sample PostgreSQL linstor.toml looks like this:

A sample MariaDB linstor.toml looks like this:

etcd is a distributed key-value store that makes it easy to keep your LINSTOR database distributed in a HA-setup. The etcd driver is already included in the LINSTOR-controller package and only needs to be configured in the linstor.toml.

More information about how to install and configure etcd can be found here: etcd docs

And here is a sample [db] section from the linstor.toml:

To make LINSTOR’s administrative tasks more accessible and also available for web-frontends a REST API has been created. The REST API is embedded in the linstor-controller and since LINSTOR 0.9.13 configured through the linstor.toml configuration file.

If you want to use the REST API the current documentation can be found on the following link: https://app.swaggerhub.com/apis-docs/Linstor/Linstor/

The HTTP REST API can also run secured by HTTPS and is highly recommended if you use any features that require authorization. To do so you have to create a Java keystore file with a valid certificate that will be used to encrypt all HTTPS traffic.

Here is a simple example on how you can create a self signed certificate with the keytool that is included in the Java Runtime:

keytool will ask for a password to secure the generated keystore file and is needed for the LINSTOR Controller configuration. In your linstor.toml file you have to add the following section:

Now (re)start the linstor-controller and the HTTPS REST API should be available on port 3371.

More information about how to import other certificates can be found here: https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html

The LINSTOR-Controller also provides a /health HTTP path that will simply return HTTP-Status 200 if the controller can access its database and all services are up and running. Otherwise it will return HTTP error status code 500 Internal Server Error.

After configuring the LINSTOR Controller to authenticate users through LDAP (or LDAPS), and the LINSTOR REST API HTTPS, you will need to enter LINSTOR commands as follows:

If you have configured LDAP authentication without also configuring LINSTOR REST API HTTPS, you will need to explicitly enable password authentication over HTTP, by using the --allow-insecure-path flag with your linstor commands. This is not recommended outside of a secured and isolated LAN, as you will be sending credentials in plain text.

The LINSTOR Controller will prompt you for the user’s password, in each of the above examples. You may optionally use the --password argument to supply the user’s password on the command line, with all the warnings of caution that would go along with doing so.

LINSTOR automatically configures quorum policies on resources when quorum is achievable. This means, whenever you have at least two diskful and one or more diskless resource assignments, or three or more diskful resource assignments, LINSTOR will enable quorum policies for your resources automatically.

Inversely, LINSTOR will automatically disable quorum policies whenever there are less than the minimum required resource assignments to achieve quorum.

This is controlled through the, DrbdOptions/auto-quorum, property which can be applied to the linstor-controller, resource-group, and resource-definition. Accepted values for the DrbdOptions/auto-quorum property are disabled, suspend-io, and io-error.

Setting the DrbdOptions/auto-quorum property to disabled will allow you to manually, or more granularly, control the quorum policies of your resources should you want to.

For example, to manually set the quorum policies of a resource-group named my_ssd_group, you would use the following commands:

You may want to disable DRBD’s quorum features completely. To do that, you would need to first disable DrbdOptions/auto-quorum on the appropriate LINSTOR object, and then set the DRBD quorum features accordingly. For example, use the following commands to disable quorum entirely on the my_ssd_group resource-group:

If a satellite is offline for a prolonged period of time, LINSTOR can be configured to declare that node as evicted. This triggers an automated reassignment of the affected DRBD resources to other nodes to ensure a minimum replica count is kept.

This feature uses the following properties to adapt the behaviour.

After the linstor-controller loses the connection to a satellite, aside from trying to reconnect, it starts a timer for that satellite. As soon as that timer exceeds DrbdOptions/AutoEvictAfterTime and all of the DRBD-connections to the DRBD-resources on that satellite are broken, the controller will check whether or not DrbdOptions/AutoEvictMaxDisconnectedNodes has been met. If it hasn’t, and DrbdOptions/AutoEvictAllowEviction is true for the node in question, the satellite will be marked as EVICTED. At the same time, the controller will check for every DRBD-resource whether the number of resources is still above DrbdOptions/AutoEvictMinReplicaCount. If it is, the resource in question will be marked as DELETED. If it isn’t, an auto-place with the settings from the corresponding resource-group will be started. Should the auto-place fail, the controller will try again later when changes that might allow a different result, such as adding a new node, have happened. Resources where an auto-place is necessary will only be marked as DELETED if the corresponding auto-place was successful.

The evicted satellite itself will not be able to reestablish connection with the controller. Even if the node is up and running, a manual reconnect will fail. It is also not possible to delete the satellite, even if it is working as it should be. The satellite can, however, be restored. This will remove the EVICTED-flag from the satellite and allow you to use it again. Previously configured network interfaces, storage pools, properties and similar entities as well as non-DRBD-related resources and resources that could not be autoplaced somewhere else will still be on the satellite. To restore a satellite, use

Should you want to instead throw everything that once was on that node, including the node itself, away, you need to use the node lost command instead.

You can set the LINSTOR auto-diskful and auto-diskful-allow-cleanup properties for various LINSTOR objects, for example, a resource definition, to have LINSTOR help automatically make a Diskless node Diskful and perform appropriate cleanup actions afterwards.

This is useful when a Diskless node has been in a Primary state for a DRBD resource for more than a specified number of minutes. This could happen in cases where you integrate LINSTOR managed storage with other orchestrating and scheduling platforms, such as OpenStack, OpenNebula, and others. On some platforms that you integrate LINSTOR with, you might not have a way to influence where in your cluster a storage volume will be used.

The auto-diskful options give you a way to use LINSTOR to sensibly delegate the roles of your storage nodes in response to an integrated platform’s actions that are beyond your control.

These LINSTOR properties may be set using the set-property command and may be set on the following objects: volume, storage pool, resource, controller, or node. You can also set these QoS properties on resource groups, volume groups, resource definitions, or volume definitions. When you set a QoS property on a group or definition, resources created from the group or definition will inherit the QoS settings.

The following example shows creating a resource group, then creating a volume group, then applying QoS settings to the volume group, and then spawning resources from the resource group. A verification command will show that the spawned resources inherit the QoS settings. The example uses an assumed previously created LINSTOR storage pool named pool1. You will need to replace this name with a storage pool name that exists in your environment.

To verify that the spawned resources inherited the QoS setting, you can show the contents of the corresponding sysfs file, on a node that contributes storage to the storage pool.

A single LINSTOR volume can be composed of multiple DRBD devices. For example, DRBD with external metadata will have three backing devices: a data (storage) device, a metadata device, and the composite DRBD device (volume) provided to LINSTOR. If the data and metadata devices correspond to different backing disks, then if you set a sysfs property for such a LINSTOR volume, only the local data (storage) backing device will receive the property value in the corresponding /sys/fs/cgroup/blkio/ file. Neither the device backing DRBD’s metadata, nor the composite backing device provided to LINSTOR would receive the value. However, when DRBD’s data and its metadata share the same backing disk, QoS settings will affect the performance of both data and metadata operations.

In case a LINSTOR resource definition has an nvme-target as well as an nvme-initiator resource, both data (storage) backing devices of each node will receive the sysfs property value. In case of the target, the data backing device will be the volume of either LVM or ZFS, whereas in case of the initiator, the data backing device will be the connected nvme-device, regardless of which other LINSTOR layers, such as LUKS, NVMe, DRBD, and others (see Using LINSTOR Without DRBD), are above that.

A quick way to list available commands on the command line is to type linstor.

Further information about subcommands (e.g., list-nodes) can be retrieved in two ways:

Using the ‘help’ subcommand is especially helpful when LINSTOR is executed in interactive mode (linstor interactive).

One of the most helpful features of LINSTOR is its rich tab-completion, which can be used to complete basically every object LINSTOR knows about (e.g., node names, IP addresses, resource names, …​). In the following examples, we show some possible completions, and their results:

If tab-completion does not work out of the box, please try to source the appropriate file:

For zsh shell users, the linstor-client command can generate a zsh compilation file, that has basic support for command and argument completion.

If something goes wrong and you need help finding the cause of the issue, you can use

The command above will create a new sos-report in /var/log/linstor/controller/ on the controller node. Alternatively you can use

which will create a new sos-report and additionally downloads that report to the local machine into your current working directory.

This sos-report contains logs and useful debug-information from several sources (Linstor-logs, dmesg, versions of external tools used by LINSTOR, ip a, database dump and many more). These information are stored for each node in plain text in the resulting .tar.gz file.

For help from the community please subscribe to our mailing list located here: https://lists.linbit.com/listinfo/drbd-user

To file bug or feature request please check out our GitHub page https://github.com/linbit

Alternatively, if you need to purchase remote installation services, 24/7 support, access to certified repositories, or feature development, please contact us: +1-877-454-6248 (1-877-4LINBIT) , International: +43-1-8178292-0 | [email protected]

To deploy the Operator, create a kustomization.yaml file. This will declare your pull secret for drbd.io and allow you to pull in the Operator deployment. The Operator will be deployed in a new namespace linbit-sds. Make sure to replace MY_LINBIT_USER and MY_LINBIT_PASSWORD with your own credentials. You can find the latest releases on charts.linstor.io.

Then, apply the kustomization.yaml file, by using kubectl command, and wait for the Operator to start:

The Operator is now ready to deploy LINBIT SDS for Kubernetes.

Deploying LINBIT SDS for Kubernetes with the Operator is as simple as creating a new LinstorCluster resource and waiting for the Operator to complete the setup:

By default, LINBIT SDS for Kubernetes does not configure any storage. To add storage, you can configure a LinstorSatelliteConfiguration, which the Operator uses to configure one or more satellites.

The following example creates a simple FILE_THIN pool and it does not require any additional set up on the host:

Other types of storage pools can be configured as well. Refer to the examples upstream.

By configuring key and certificate based encryption, you can make communication between certain LINSTOR components, for example, between LINSTOR satellite nodes and a LINSTOR controller node, or between the LINSTOR client and the LINSTOR API, more secure.

Within LINSTOR Operator v2 deployments, you can change the cluster state by modifying LINSTOR related Kubernetes CustomResourceDefinitions (CRDs) or check the status of a resource. An overview list of these resources follows. Refer to the upstream Piraeus project’s API reference (linked for each resource below) for more details.

After deploying LINBIT SDS for Kubernetes, you can continue with the Getting Started with LINBIT SDS Storage in Kubernetes, Configuring the DRBD Module Loader in Operator v2 Deployments, Using the Host Network for DRBD Replication in Operator v2 Deployments sections in this chapter, or refer to the available tutorials in the upstream Piraeus project.

The LINSTOR controller can use the Kubernetes API directly to persist its cluster state. To enable this back end, use the following override file during the chart installation:

You can use the pv-hostpath Helm templates to create hostPath persistent volumes. Create as many PVs as needed to satisfy your configured etcd replicas (default 1).

Create the hostPath persistent volumes, substituting cluster node names accordingly in the nodes= option:

By default, a PV is created on every control-plane node. You can manually select the storage nodes by passing --set "nodes={<NODE0>,<NODE1>,<NODE2>}" to the install command.

LINSTOR can connect to an existing PostgreSQL, MariaDB or etcd database. For instance, for a PostgreSQL instance with the following configuration:

The Helm chart can be configured to use this database rather than deploying an etcd cluster, by adding the following to the Helm install command:

The LINSTOR Operator v1 can automate some basic storage set up for LINSTOR.

This section describes the different options for enabling security features available when using a LINSTOR Operator v1 deployment (using Helm) in Kubernetes.

The default communication between LINSTOR components is not secured by TLS. If this is needed for your setup, choose one of three methods:

All the below examples use the following sp-values.yaml file. Feel free to adjust this for your uses and environment. See [Configuring storage pool creation] for further details.

Install with LINSTOR storage-pools defined at install through sp-values.yaml, persistent hostPath volumes, three etcd replicas, and by compiling the DRBD kernel modules for the host kernels.

This should be adequate for most basic deployments. Note that this deployment is not using the pre-compiled DRBD kernel modules just to make this command more portable. Using the pre-compiled binaries will make for a much faster install and deployment. Using the Compile option would not be suggested for use in a large Kubernetes clusters.

Install with LINSTOR storage-pools defined at install through sp-values.yaml, use an already created PostgreSQL DB (preferably clustered), rather than etcd, and use already compiled kernel modules for DRBD.

The PostgreSQL database in this particular example is reachable through a service endpoint named postgres. PostgreSQL itself is configured with POSTGRES_DB=postgresdb, POSTGRES_USER=postgresadmin, and POSTGRES_PASSWORD=admin123

To protect the storage infrastructure of the cluster from accidentally deleting vital components, it is necessary to perform some manual steps before deleting a Helm deployment.

The Helm charts provide a set of further customization options for advanced use cases.

To create a high-availability deployment of all components within a LINSTOR Operator v1 deployment, consult the upstream guide The default values are chosen so that scaling the components to multiple replicas ensures that the replicas are placed on different nodes. This ensures that a single node failures will not interrupt the service.

To create a backup of the etcd database (in LINSTOR Operator v1 deployments) and store it on your control host, enter the following commands:

These commands will create a file save.db on the machine you are running kubectl from.

The instructions in this section describe how you can connect an Operator v2 LINBIT SDS deployment to an existing LINBIST SDS cluster that you manage outside Kubernetes.

To follow the steps in this section you should be familiar with editing LinstorCluster resources.

To skip the creation of a LINSTOR controller deployment and configure the other components to use your existing LINSTOR controller, use the following options when running helm install:

After all pods are ready, you should see the Kubernetes cluster nodes as satellites in your LINSTOR setup.

Create a storage class referencing an existing storage pool on your storage nodes.

You can provision new volumes by creating PVCs using your storage class. The volumes will first be placed only on nodes with the given storage pool, that is, your storage infrastructure. Once you want to use the volume in a pod, LINSTOR CSI will create a diskless resource on the Kubernetes node and attach over the network to the diskful resource.

To simplify entering LINSTOR client commands within a Kubernetes deployment, you can use the kubectl-linstor utility. This utility is available from the upstream Piraeus datastore project. To download it, enter the following commands on your Kubernetes control plane node:

To install the utility, first extract it and then move the extracted executable file to a directory in your $PATH, for example, /usr/bin. Then you can use kubectl-linstor to get access to the complete LINSTOR CLI.

It also expands references to PVCs to the matching LINSTOR resource.

It also expands references of the form pod:[<namespace>/]<podname> into a list resources in use by the pod.

This should only be necessary for investigating problems and accessing advanced functionality. Regular operation such as creating volumes should be achieved through the Kubernetes integration.

The following storage class contains all currently available parameters to configure the provisioned storage.

The csi.storage.k8s.io/fstype parameter sets the file system type to create for volumeMode: FileSystem PVCs. Currently supported are:

autoPlace is an integer that determines the amount of replicas a volume of this StorageClass will have. For instance, autoPlace: "3" will produce volumes with three-way replication. If neither autoPlace nor nodeList are set, volumes will be automatically placed on one node.

placementCount is an alias for autoPlace

The LINSTOR Resource Group (RG) to associate with this StorageClass. If not set, a new RG will be created for each new PVC.

storagePool is the name of the LINSTOR storage pool that will be used to provide storage to the newly-created volumes.

disklessStoragePool is an optional parameter that only affects LINSTOR volumes that are assigned as “diskless” to kubelets, that is, as clients. If you have a custom diskless storage pool defined in LINSTOR, you will specify that here.

A comma-separated list of layers to use for the created volumes. The available layers and their order are described towards the end of this section. Defaults to drbd,storage

Select from one of the available volume schedulers:

Control on which nodes a volume is accessible. The value for this option can take two different forms:

encryption is an optional parameter that determines whether to encrypt volumes. LINSTOR must be configured for encryption for this to work properly.

nodeList is a list of nodes for volumes to be assigned to. This will assign the volume to each node and it will be replicated among all of them. This can also be used to select a single node by hostname, but it’s more flexible to use replicasOnSame to select a single node.

clientList is a list of nodes for diskless volumes to be assigned to. Use in conjunction with nodeList.

replicasOnSame is a list of key or key=value items used as auto-placement selection labels when autoPlace is used to determine where to provision storage. These labels correspond to LINSTOR node properties.

Let’s explore this behavior with examples assuming a LINSTOR cluster such that node-a is configured with the following auxiliary property zone=z1 and role=backups, while node-b is configured with only zone=z1.

If we configure a StorageClass with autoPlace: "1" and replicasOnSame: "zone=z1 role=backups", then all volumes created from that StorageClass will be provisioned on node-a, since that is the only node with all of the correct key=value pairs in the LINSTOR cluster. This is the most flexible way to select a single node for provisioning.

If we configure a StorageClass with autoPlace: "1" and replicasOnSame: "zone=z1", then volumes will be provisioned on either node-a or node-b as they both have the zone=z1 aux prop.

If we configure a StorageClass with autoPlace: "2" and replicasOnSame: "zone=z1 role=backups", then provisioning will fail, as there are not two or more nodes that have the appropriate auxiliary properties.

If we configure a StorageClass with autoPlace: "2" and replicasOnSame: "zone=z1", then volumes will be provisioned on both node-a and node-b as they both have the zone=z1 aux prop.

You can also use a property key without providing a value to ensure all replicas are placed on nodes with the same property value, with caring about the particular value. Assuming there are 4 nodes, node-a1 and node-a2 are configured with zone=a. node-b1 and node-b2 are configured with zone=b. Using autoPlace: "2" and replicasOnSame: "zone" will place on either node-a1 and node-a2 OR on node-b1 and node-b2.

replicasOnDifferent takes a list of properties to consider, same as replicasOnSame. There are two modes of using replicasOnDifferent:

Create a diskless resource on all nodes that were not assigned a diskful resource.

Do not place the resource on a node which has a resource with a name matching the regular expression.

fsOpts is an optional parameter that passes options to the volume’s file system at creation time.

mountOpts is an optional parameter that passes options to the volume’s file system at mount time.

Extra arguments to pass to xfs_io, which gets called before right before first use of the volume.

Parameters starting with property.linstor.csi.linbit.com/ are translated to LINSTOR properties that are set on the Resource Group associated with the StorageClass.

For example, to set DrbdOptions/auto-quorum to disabled, use:

The full list of options is available here

Advanced DRBD options to pass to LINSTOR. For example, to change the replication protocol, use DrbdOptions/Net/protocol: "A".

Before you can add snapshot support within a LINBIT SDS deployment, you need to meet the following environment prerequisites:

LINSTOR can store snapshots on S3 compatible storage for disaster recovery. This is integrated in Kubernetes using a special VolumeSnapshotClass:

Check the LINSTOR snapshot guide on the exact meaning of the snap.linstor.csi.linbit.com/ parameters. The credentials used to log in are stored in a separate secret, as show in the example above.

Referencing the above storage class when creating snapshots causes the snapshots to be automatically uploaded to the configured S3 storage.

The following example show common scenarios where you want to optimize volume accessibility and locality. It also includes examples of how to spread volume replicas across zones in a cluster.

In some circumstances it might be necessary to disable the DRBD module loader entirely. For example, if you are using an immutable operating system, and DRBD and other modules are loaded as part of the host configuration.

To disable the DRBD module loader completely, apply the following YAML configuration to your deployment:

By default, the Operator will try to find a DRBD module loader that matches the host operating system. The Operator determines the host distribution by inspecting the .status.nodeInfo.osImage field of the Kubernetes Node resource. A user-defined image can be used if the automatic mapping does not succeed or if you have different module loading requirements.

The following YAML configuration overrides the chosen DRBD module loader image with a user-defined image example.com/drbd-loader:v9:

drbd.io, available to LINBIT customers only, maintains the following module loader container images:

If you need to create a module loader image for your own distribution, you can refer to the container source files which are available in the upstream Piraeus project.

By default, the DRBD module loader will try to build the kernel module from source. The module loader can also be configured to load the module from a DEB or RPM package included in the image, or skip loading DRBD entirely.

To change the behavior of the DRBD module loader, set the LB_HOW environment variable to an appropriate value shown in the following table:

After setting the LB_HOW environment variable, apply the following YAML configuration to your deployment. Based on the name within the metadata section, the example below would be used with an LB_HOW environment variable that was set to deps_only.

Switching from the default container network to the host network for DRBD replication is possible at any time. Existing DRBD resources will then be reconfigured to use the host network interface.

To configure the host network for the LINSTOR satellite, apply the following YAML configuration to your deployment:

After the satellite pods are recreated, they will use the host network. Any existing DRBD resources are reconfigured to use a new IP address on the host network rather than an IP address on the container network.

Switching back from host network to container network involves manually resetting the configured peer addresses used by DRBD. You can do this by rebooting every node, or by manually resetting the addresses by using the drbdadm CLI command on each node. Each method is described below.

[[s-kubernetes-drbd-replication-switching-from-host-to-container-network-node-rebooting-v2 ===== Rebooting Nodes to Switch DRBD Replication from the Host to the Container Network

First, you need to remove the LinstorSatelliteConfiguration that set hostNetwork: true. You can do this by entering the following kubectl command:

Next, reboot each cluster node, either serially, one by one, or else all at once. In general, replication will not work between rebooted nodes and non-rebooted nodes. The non-rebooted nodes will continue to use the host network addresses, which are generally not reachable from the container network.

After all nodes have restarted, all resources will be configured to use the container network, and all DRBD connections should be connected again.

[[s-kubernetes-drbd-replication-switching-from-host-to-container-network-node-drbdadm-v2 ===== Using the DRBD Administration Tool to Switch DRBD Replication from the Host to the Container Network

First, you need to temporarily stop all DRBD replication and suspend all DRBD volume I/O operations by using the drbdadm suspend-io all command. Enter the command once on each LINSTOR satellite pod.

Next, disconnect all DRBD connections on all nodes.

Next, you can safely reset all DRBD connection paths. This frees the connection on each node to be moved to the container network.

Finally, remove the LinstorSatelliteConfiguration resource configuration that set hostNetwork: true. This will result in the creation of new LINSTOR satellite pods that use the container network.

After the pods are recreated and the LINSTOR satellites are Online, the DRBD resource will be reconfigured and resume I/O operations.

In Operator v1 deployments, the operator will set up monitoring containers along the existing components and make them available as a Service.

If you use the Prometheus Operator, the LINSTOR Operator will also set up the ServiceMonitor instances. The metrics will automatically be collected by the Prometheus instance associated to the operator, assuming watching the Piraeus namespace is enabled.

To disable exporting of metrics, set operator.satelliteSet.monitoringImage to an empty value.

If you deployed LINSTOR using Operator v2, you can upgrade LINBIT SDS by deploying the newer version of LINSTOR Operator.

To upgrade, change the tag referenced in the kustomization file:

Then, apply the kustomization.yaml file, by using the following kubectl command, and wait for the upgrade to complete:

Before upgrading to a new release, you should ensure you have an up-to-date backup of the LINSTOR database. If you are using the etcd database packaged in the LINSTOR Chart, see here

Upgrades will update to new versions of the following components:

Some versions require special steps, refer to instructions here The main command to upgrade to a new LINSTOR Operator version is:

If you used any customizations on the initial install, pass the same options to helm upgrade. The options currently in use can be retrieved from Helm.

This triggers the rollout of new pods. After a short wait, all pods should be running and ready. Check that no errors are listed in the status section of LinstorControllers, LinstorSatelliteSets and LinstorCSIDrivers.

The LINSTOR Operator will create a backup of the database before upgrading. This makes it possible to roll back to a known state should something go wrong during the upgrade.

There are situations in which the Operator can’t create the backup automatically. This might be because:

Some versions require special steps, see below.

The next steps are exactly the same as in Kubernetes:

The LINSTOR container images come with the LINBIT GUI preinstalled. To expose it on your cluster, configure an OpenShift Route resource:

The GUI is now accessible at https://linstor-op-cs-storage.apps.oc.example.com/ui/

OpenShift includes a fully configured monitoring stack. While most of the monitoring stack is only intended for OpenStack infrastructure, basic monitoring for LINBIT SDS is possible.

First, ensure that monitoring of user-defined projects is enabled in OpenShift by following the steps in Red Hat documentation.

Once monitoring for user-defined projects is enabled, the LINSTOR Operator automatically detects the presence of OpenShift’s Prometheus-based monitoring stack and configures the ServiceMonitor resources. The Prometheus instance will scrape metrics for DRBD and LINSTOR from all Cluster nodes.

To run LINSTOR, every Nomad agent needs to be configured to:

The LINSTOR Controller is deployed as a service with no replicas. At any point in time, there can only be one LINSTOR Controller running in a cluster. It is possible to restart the controller on a new node, provided it still has access to the database. See Creating a Highly Available LINSTOR Cluster for more information.

The following example will create a Nomad job starting a single LINSTOR Controller in datacenter dc1 and connect to an external database.

Apply the job by running:

In Nomad, the LINSTOR satellites are deployed as a system job that runs in a privileged container. In addition to the satellites, the job will also load the DRBD module along with other kernel modules used by LINSTOR.

The following example will create a Nomad job starting a LINSTOR satellite on every node in data center dc1.

Apply the job by running:

Once the linstor-controller and linstor-satellite jobs are running, you can start configuring the cluster using the linstor command line tool.

This can done:

In all cases, you need to add the satellites to your cluster and create some storage pools. For example, to add the node nomad-01.example.com and configure a LVM Thin storage pool, you would run: