Objects are the end result which LINSTOR presents to the end-user or application, such as: Kubernetes/OpenShift, a replicated block device (DRBD), NVMeOF target, and others.

To use LINSTOR to create storage volumes, you will need to install a volume manager, either LVM or ZFS, if one is not already installed on your system.

If you are a LINBIT customer, you can download a LINBIT created helper script and run it on your nodes to:

Enabling LINBIT package repositories will give you access to LINBT software packages, DRBD kernel modules, and other related software such as cluster managers and OCF scripts. You can then use a package manager to fetch, install, and manage updating installed packages.

After registering your node and enabling the drbd-9 LINBIT package repository, you can use a DEB, RPM, or YaST2 based package manager to install LINSTOR and related components.

The LINSTOR project’s GitHub page is here: https://github.com/LINBIT/linstor-server.

LINBIT also has downloadable archived files of source code for LINSTOR, DRBD, and more, available here: https://linbit.com/linbit-software-download-page-for-linstor-and-drbd-linux-driver/.

In clusters where you have only one kind of storage and the capability to hot swap storage devices, you may choose a model where you create one storage pool per physical backing device. The advantage of this model is to confine failure domains to a single storage device.

Both the Exos and LVM2 storage providers offer the option of multiple server nodes directly connected to the storage array and drives. With LVM2 the external locking service (lvmlockd) manages volume groups created with the –shared options with vgcreate. The --shared-space can be used when configuring a LINSTOR pool to use the same LVM2 volume group accessible by two or more nodes. The example below shows using the LVM2 volume group UUID as the shared space identifier for a pool accessible by nodes alpha and bravo:

Exos pools will use the Exos pool serial number by default for the shared-space identifier.

Since linstor-server 1.5.2 and a recent linstor-client, LINSTOR can create LVM/ZFS pools on a satellite for you. The linstor-client has the following commands to list possible disks and create storage pools, but such LVM/ZFS pools are not managed by LINSTOR and there is no delete command, so such action must be done manually on the nodes.

Will give you a list of available disks grouped by size and rotational(SSD/Magnetic Disk).

It will only show disks that pass the following filters:

With the create-device-pool command you can create a LVM pool on a disk and also directly add it as a storage-pool in LINSTOR.

If the --storage-pool option was provided, LINSTOR will create a storage-pool with the given name.

For more options and exact command usage please check the linstor-client help.

LINSTOR has the following supported storage plug-ins at the time of writing:

使用 resource create 命令，可以将资源定义显式分配给命名节点。

It is possible to have LINSTOR select nodes and storage pools to deploy a resource to, by using the linstor resource create --auto-place or linstor resource-definition auto-place commmands. This section will use the resource create --auto-place command in examples. However, you can use either command to produce the same results.

In the following example, the value after the --auto-place option tells LINSTOR how many replicas you want to have. The storage-pool option should be obvious.

可能不太明显的是，您可以省略 --storage-pool 选项，然后LINSTOR可以自己选择一个存储池。选择遵循以下规则：

The remaining storage pools will be rated by different strategies. LINSTOR has currently three strategies:

MaxFreeSpace: This strategy maps the rating 1:1 to the remaining free space of the storage pool. However, this strategy only considers the actually allocated space (in case of thin-provisioned storage pool this might grow with time without creating new resources) MinReservedSpace: Unlike the “MaxFreeSpace”, this strategy considers the reserved space. That is the space that a thin volume can grow to before reaching its limit. The sum of reserved spaces might exceed the storage pool’s capacity, which is as overprovisioning. MinRscCount: Simply the count of resources already deployed in a given storage pool MaxThroughput: For this strategy, the storage pool’s Autoplacer/MaxThroughput property is the base of the score, or 0 if the property is not present. Every Volume deployed in the given storage pool will subtract its defined sys/fs/blkio_throttle_read and sys/fs/blkio_throttle_write property- value from the storage pool’s max throughput. The resulting score might be negative.

The scores of the strategies will be normalized, weighted and summed up, where the scores of minimizing strategies will be converted first to allow an overall maximization of the resulting score.

The weights of the strategies can be configured with the following command:

The strategy names are listed above and the weight can be an arbitrary decimal.

Finally, LINSTOR tries to find the best matching group of storage pools meeting all requirements. This step also considers other auto-placement restrictions such as --replicas-on-same, --replicas-on-different, --do-not-place-with, --do-not-place-with-regex, --layer-list, and --providers.

You can delete a resource definition by using the command:

This will remove the named resource definition from the entire LINSTOR cluster. The resource is removed from all nodes and the resource entry is marked for removal from LINSTOR’s database tables. After LINSTOR has removed the resource from all the nodes, the resource entry is removed from LINSTOR’s database tables.

You can delete a resource using the command:

Unlike deleting a resource definition, this command will only delete a LINSTOR resource from the node (or nodes) that you specify. The resource is removed from the node and the resource entry is marked for removal from LINSTOR’s database tables. After LINSTOR has removed the resource from the node, the resource entry is removed from LINSTOR’s database tables.

Deleting a LINSTOR resource may have implications for a cluster, beyond just removing the resource. For example, if the resource is backed by a DRBD layer, removing a resource from one node in a three node cluster could also remove certain quorum related DRBD options, if any existed for the resource. After removing such a resource from a node in a three node cluster, the resource would no longer have quorum as it would now only be deployed on two nodes in the three node cluster.

After running a linstor resource delete command to remove a resource from a single node, you might see informational messages such as:

Also unlike deleting a resource definition, you can delete a resource while there are existing snapshots of the resource’s storage pool. Any existing snapshots for the resource’s storage pool will persist.

You can delete a resource group by using the command:

As you might expect, this command deletes the named resource group. You can only delete a resource group if it has no associated resource definitions, otherwise LINSTOR will present an error message, such as:

To resolve this error so that you can delete the resource group, you can either delete the associated resource definitions, or your can move the resource definitions to another (existing) resource group:

You can find which resource definitions are associated with your resource group by entering the following command:

For configuring the highly-available storage we use LINSTOR itself. This has the advantage that the storage is under LINSTOR control and can for example be easily extended to new cluster nodes. Just create a new resource with 200MB in size. This could look like this, you certainly need to adapt the storage pool name:

From now on we assume the resource’s name is “linstor_db”. It is crucial that your cluster qualifies for auto-quorum and uses the io-error policy (see Section Auto-Quorum Policies), and that auto-promote is disabled.

After the resource is created, it is time to move the LINSTOR DB to the new storage and to create a systemd mount service. First we stop the current controller and disable it, as it will be managed by drbd-reactor later.

Copy the /etc/systemd/system/var-lib-linstor.mount mount file to all the standby nodes for the linstor controller. Again, do not systemctl enable any of these services, they get managed by drbd-reactor.

The next step is to install LINSTOR controllers on all nodes that have access to the linstor_db DRBD resource (as they need to mount the DRBD volume) and which you want to become a possible LINSTOR controller. It is important that the controllers are manged by drbd-reactor, so verify that the linstor-controller.service is disabled on all nodes! To be sure, execute systemctl disable linstor-controller on all cluster nodes and systemctl stop linstor-controller on all nodes except the one it is currently running from the previous step. Also verify that you have set chattr +i /var/lib/linstor on all potential controller nodes if you use LINSTOR version equal or greater to 1.14.0.

For starting and stopping the mount service and the linstor-controller service we use drbd-reactor. Install this component on all nodes that could become a LINSTOR controller and edit their /etc/drbd-reactor.d/linstor_db.toml configuration file. It should contain an enabled promoter plug-in section like this:

Depending on your requirements you might also want to set an on-stop-failure action and set stop-services-on-exit.

After that restart drbd-reactor and enable it on all the nodes you configured it.

Check that there are no warnings from drbd-reactor service in the logs by running systemctl status drbd-reactor. As there is already an active LINSTOR controller things will just stay the way they are. Run drbd-reactorctl status linstor_db to check the health of the linstor_db target unit.

The last but nevertheless important step is to configure the LINSTOR satellite services to not delete (and then regenerate) the resource file for the LINSTOR controller DB at its startup. Do not edit the service files directly, but use systemctl edit. Edit the service file on all nodes that could become a LINSTOR controller and that are also LINSTOR satellites.

After this change you should execute systemctl restart linstor-satellite on all satellite nodes.

NVMe-oF/NVMe-TCP允许LINSTOR将无盘资源连接到具有相同资源的节点，数据存储在NVMe结构上。这就带来了这样一个优势：通过网络访问数据，无需使用本地存储就可以装载资源。在这种情况下，LINSTOR不使用DRBD，因此不会复制LINSTOR提供的NVMe资源，数据存储在一个节点上。

要将NVMe-oF/NVMe-TCP与LINSTOR一起使用，需要在充当Satellite的每个节点上安装包 nvme-cli，并将NVMe-oF/NVMe-TCP用于资源：

要使资源使用NVMe-oF/NVMe-TCP，必须在创建资源定义时提供一个附加参数：

To use NVMe-TCP rather than the default NVMe-oF, the following property needs to be set:

The property NVMe/TRType can alternatively be set on resource-group or controller level.

Next, create the volume-definition for our resource:

在节点上创建资源之前，必须知道数据将在本地存储在哪里，以及哪个节点通过网络访问它。

首先，我们在存储数据的节点上创建资源：

在将通过网络访问资源数据的节点上，必须将资源定义为无盘：

现在，您可以在一个节点上装载资源 nvmedata。

Since version 1.5.0 the additional Layer openflex can be used in LINSTOR. From LINSTOR’s perspective, the OpenFlex Composable Infrastructure takes the role of a combined layer acting as a storage layer (like LVM) and also providing the allocated space as an NVMe target. OpenFlex has a REST API which is also used by LINSTOR to operate with.

As OpenFlex combines concepts of LINSTOR’s storage as well as NVMe-layer, LINSTOR was added both, a new storage driver for the storage pools as well as a dedicated openflex layer which uses the mentioned REST API.

In order for LINSTOR to communicate with the OpenFlex-API, LINSTOR needs some additional properties, which can be set once on controller level to take LINSTOR-cluster wide effect:

Once that is configured, we can now create LINSTOR objects to represent the OpenFlex architecture. The theoretical mapping of LINSTOR objects to OpenFlex objects are as follows: Obviously an OpenFlex storage pool is represented by a LINSTOR storage pool. As the next thing above a LINSTOR storage pool is already the node, a LINSTOR node represents an OpenFlex storage device. The OpenFlex objects above storage device are not mapped by LINSTOR.

When using NVMe, LINSTOR was designed to run on both sides, the NVMe target as well as on the NVMe initiator side. In the case of OpenFlex, LINSTOR cannot (or even should not) run on the NVMe target side as that is completely managed by OpenFlex. As LINSTOR still needs nodes and storage pools to represent the OpenFlex counterparts, the LINSTOR client was extended with special node create commands since 1.0.14. These commands not only accept additionally needed configuration data, but also starts a “special satellite” besides the already running controller instance. These special satellites are completely LINSTOR managed. They will shut down when the controller shuts down and will be started again when the controller starts.

The new client command for creating a “special satellite” representing an OpenFlex storage device is:

The arguments are as follows:

The last step of the configuration is the creation of LINSTOR storage pools:

Once all necessary storage pools are created in LINSTOR, the next steps are similar to the usage of using an NVMe resource with LINSTOR. Here is a complete example:

A DM-Writecache device is composed of two devices: one storage device and one cache device. LINSTOR can setup such a writecache device, but needs some additional information, like the storage pool and the size of the cache device.

The two properties set in the examples are mandatory, but can also be set on controller level which would act as a default for all resources with WRITECACHE in their --layer-list. However, please note that the Writecache/PoolName refers to the corresponding node. If the node does not have a storage-pool named pmempool you will get an error message.

The 4 mandatory parameters required by DM-Writecache are either configured through a property or figured out by LINSTOR. The optional properties listed in the mentioned link can also be set through a property. Please see linstor controller set-property --help for a list of Writecache/* property-keys.

使用 --layer-list DRBD,WRITECACHE,STORAGE 将DRBD配置为使用外部元数据时，只有备份设备将使用writecache，而不是保存外部元数据的设备。

LINSTOR can also setup a DM-Cache device, which is very similar to the DM-Writecache from the previous section. The major difference is that a cache device is composed by three devices: one storage device, one cache device and one meta device. The LINSTOR properties are quite similar to those of the writecache but are located in the Cache namespace:

Please see linstor controller set-property --help for a list of Cache/* property-keys and default values for omitted properties.

Using --layer-list DRBD,CACHE,STORAGE while having DRBD configured to use external metadata, only the backing device will use a cache, not the device holding the external metadata.

The storage layer will provide new devices from well known volume managers like LVM, ZFS or others. Every layer combination needs to be based on a storage layer, even if the resource should be diskless – for that type there is a dedicated diskless provider type.

For a list of providers with their properties please see Storage Providers.

For some storage providers LINSTOR has special properties:

A storage pool with the type remote SPDK can only be created on a “special satellite”. For this you first need to start a new satellite using the command:

This will start a new satellite instance running on the same machine as the controller. This special satellite will do all the REST based RPC communication towards the remote SPDK proxy. As the help message of the LINSTOR command shows, the administrator might want to use additional settings when creating this special satellite:

The difference between the --api-* and their corresponding --api-\*-env versions is that the version with the -env ending will look for an environment variable containing the actual value to use whereas the --api-\* version directly take the value which is stored in the LINSTOR property. Administrators might not want to save the --api-pw in plain text, which would be clearly visible using commands like linstor node list-property <nodeName>.

Once that special satellite is up and running the actual storage pool can be created:

Whereas node_name is self-explanatory, name is the name of the LINSTOR storage pool and driver_pool_name refers to the SPDK logical volume store.

Once this remotespdk storage pool is created the remaining procedure is quite similar as using NVMe: First the target has to be created by creating a simple “diskful” resource followed by a second resource having the --nvme-initiator option enabled.

To use multiple network paths for DRBD setups, the PrefNic property is not sufficient. Instead the linstor node interface and linstor resource-connection path commands should be used, as shown below.

In the example above we define two network interfaces (nic1 and nic2) for each node. The last two commands create network path entries in the generated DRBD .res file. This is the relevant part of the resulting .res file:

下面是有关命令的详细信息。

在LINSTOR可以加密任何卷之前，需要创建主密码。这可以通过linstor客户端完成。

`crypt create-passphrase`将等待用户输入初始主密码（因为所有其他crypt命令都没有参数）。

如果您想更改主密码，可以使用以下方法：

luks 层可以在创建资源定义或资源本身时添加，而建议使用前一种方法，因为它将自动应用于从该资源定义创建的所有资源。

要输入主密码（在controller重新启动后），请使用以下命令：

It is possible to automate the process of creating and re-entering the master passphrase.

To use this, either an environment variable called MASTER_PASSPHRASE or an entry in /etc/linstor/linstor.toml containing the master passphrase has to be created.

The required linstor.toml looks like this:

If either one of these is set, then every time the controller starts it will check whether a master passphrase already exists. If there is none, it will create a new master passphrase as specified. Otherwise, the controller enters the passphrase.

Some evacuation cases may need special planning. For example, if you are evacuating more than one node, you can exclude the nodes from participating in LINSTOR’s resource autoplacer. You can do this by using the following command on each node that you want to evacuate:

This ensures that LINSTOR will not place resources from a node that you are evacuating onto another node that you plan on evacuating.

If you already ran a node evacuate command that has either completed or still has resources in an “Evacuating” state, you can remove the “Evacuating” state from a node by using the node restore command. This will work so long as you have not yet run a node delete command.

After restoring the node, you should use the node set-property <node_name> AutoplaceTarget true command, if you previously set the AutoplaceTarget property to “false”. This way, LINSTOR can again place resources onto the node automatically, to fulfill placement count properties that you might have set for resources in your cluster.

假设在某些节点上放置了名为 resource1 的资源定义，则可以按如下方式创建快照：

这将在资源所在的所有节点上创建快照。LINSTOR将确保即使在资源处于活动使用状态时也创建一致的快照。

Setting the resource-definition property AutoSnapshot/RunEvery LINSTOR will automatically create snapshots every X minute. The optional property AutoSnapshot/Keep can be used to clean-up old snapshots which were created automatically. No manually created snapshot will be cleaned-up / deleted. If AutoSnapshot/Keep is omitted (or ⇐ 0), LINSTOR will keep the last 10 snapshots by default.

以下步骤将快照还原到新资源。即使原始资源已从创建快照的节点中删除，也可能发生这种情况。

首先使用与快照中的卷匹配的卷定义新资源：

此时，如果需要，可以应用其他配置。然后，准备好后，根据快照创建资源：

这将在快照所在的所有节点上放置新资源。也可以显式选择放置资源的节点；请参阅帮助（ linstor snapshot resource restore -h ).

LINSTOR可以将资源回滚到快照状态。回滚时资源不能被使用。也就是说，它不能安装在任何节点上。如果资源正在使用中，请考虑是否可以通过restoring the snapshot来实现您的目标。

回滚执行如下：

资源只能回滚到最近的快照。要回滚到旧快照，请首先删除中间快照。

可以按如下方式删除现有快照：

Snapshots can be shipped between LINSTOR nodes or between different LINSTOR clusters, as well as to an S3 storage such as Amazon S3 or min.io.

The following tools need to be installed on the satellites that are going to send or receive snapshots:

You create a backup shipping schedule by using the LINSTOR client schedule create command and defining the frequency of backup shipping using cron syntax. You also need to set options that name the schedule and define various aspects of the backup shipping, such as on-failure actions, the number of local and remote backup copies to keep, and whether to also schedule incremental backup shipping.

At a minimum, the command needs a schedule name and a full backup cron schema to create a backup shipping schedule. An example command would look like this:

You can modify a backup shipping schedule by using the LINSTOR client schedule modify command. The syntax for the command is the same as that for the schedule create command. The name that you specify with the schedule modify command must be an already existing backup schedule. Any options to the command that you do not specify will retain their existing values. If you want to set the keep-local or keep-remote options back to their default values, you can set them to “all”. If you want to set the max-retries option to its default value, you can set it to “forever”.

Your physical storage is not infinite and your remote storage has a cost, so you will likely want to set limits on the number of snapshots and backups you keep.

Both the --keep-remote and --keep-local options deserve special mention as they have implications beyond what may be obvious. Using these options, you specify how many snapshots or full backups should be kept, either on the local source or the remote destination.

You can list your backup shipping schedules by using the linstor schedule list command.

For example:

The LINSTOR client schedule delete command completely deletes a backup shipping schedule LINSTOR object. The command’s only argument is the schedule name that you want to delete. If the deleted schedule is currently creating or shipping a backup, the scheduled shipping process is stopped. Depending on at which point the process stops, a snapshot, or a backup, or both, might not be created and shipped.

This command does not affect previously created snapshots or successfully shipped backups. These will be retained until they are manually deleted.

You can use the LINSTOR client backup schedule enable command to enable a previously created backup shipping schedule. The command has the following syntax:

To disable a previously enabled backup shipping schedule, you use the LINSTOR client backup schedule disable command. The command has the following syntax:

If you include the option specifying either a resource group or resource definition, as described in the backup schedule enable command example above, then you disable the schedule only for that resource group or resource definition.

For example, if you omitted specifying a resource group or resource definition in an earlier backup schedule enable command, LINSTOR would schedule backup shipping for all its deployed resources that can make snapshots. Your disable command would then only affect the resource group or resource definition that you specify with the command. The backup shipping schedule would still apply to any deployed LINSTOR resources besides the specified resource group or resource definition.

The same as for the backup schedule enable command, if you specify neither a resource group nor a resource definition, then LINSTOR disables the backup shipping schedule at the controller level for all deployed LINSTOR resources.

You can use the linstor backup schedule delete command to granularly delete either a specified resource definition or a resource group from a backup shipping schedule, without deleting the schedule itself. This command has the same syntax and arguments as the backup schedule disable command. If you specify neither a resource group nor a resource definition, the backup shipping schedule you specify will be deleted at the controller level.

It may be helpful to think about the backup schedule delete command as a way that you can remove a backup shipping schedule-remote pair from a specified LINSTOR object level, either a resource definition, a resource group, or at the controller level if neither is specified.

The backup schedule delete command does not affect previously created snapshots or successfully shipped backups. These will be retained until they are manually deleted, or until they are removed by the effects of a still applicable keep-local or keep-remote option.

You might want to use this command when you have disabled a backup schedule for multiple LINSTOR object levels and later want to affect a granular change, where a backup schedule enable command might have unintended consequences.

For example, consider a scenario where you have a backup schedule-remote pair that you enabled at a controller level. This controller has a resource group, myresgroup that has several resource definitions, resdef1 through resdef9, under it. For maintenance reasons perhaps, you disable the schedule for two resource definitions, resdef1 and resdef2. You then realize that further maintenance requires that you disable the backup shipping schedule at the resource group level, for your myresgroup resource group.

After completing some maintenance, you are able to enable the backup shipping schedule for resdef3 through resdef9, but you are not yet ready to resume (enable) backup shipping for resdef1 and resdef2. You can enable backup shipping for each resource definition individually, resdef3 through resdef9, or you can use the backup schedule delete command to delete the backup shipping schedule from the resource group, myresgroup. If you use the backup schedule delete command, backups of resdef3 through resdef9 will ship again because the backup shipping schedule is enabled at the controller level, but resdef1 and resdef2 will not ship because the backup shipping schedule is still disabled for them at the resource definition level.

When you complete your maintenance and are again ready to ship backups for resdef1 and resdef2, you can delete the backup shipping schedule for those two resource definitions to return to your starting state: backup shipping scheduled for all LINSTOR deployed resources at the controller level. To visualize this it may be helpful to refer to the decision tree diagram for how LINSTOR decides whether or not to ship a backup in the How the LINSTOR Controller Determines Scheduled Backup Shipping subsection.

You can list backup schedules by resource, using the LINSTOR client schedule list-by-resource command. This command will show LINSTOR resources and how any backup shipping schedules apply and to which remotes they are being shipped. If resources are not being shipped then the command will show:

If resources have schedule-remote-pairs and are being shipped, the command output will show when the last backup was shipped and when the next backup is scheduled to ship. It will also show whether the next and last backup shipments were full or incremental backups. Finally, the command will show when the next planned incremental (if any) and full backup shipping will occur.

You can use the --active-only flag with the schedule list-by-resource command to filter out all resources that are not being shipped.

To determine if the LINSTOR Controller will ship a deployed LINSTOR resource with a certain backup schedule for a given remote destination, the LINSTOR Controller uses the following logic:

As the diagram shows, enabled or disabled backup shipping schedules have effect in the following order:

A backup shipping schedule-remote pair that is enabled or disabled at a preceding level will override the enabled or disabled status for the same schedule-remote pair at a later level.

To determine how a LINSTOR resource will be affected by scheduled backup shipping, you can use the LINSTOR client schedule list-by-resource-details command for a specified LINSTOR resource.

The command will output a table that shows on what LINSTOR object level a backup shipping schedule is either not set (empty cell), enabled, or disabled.

By using this command, you can determine on which level you need to make a change to enable, disable, or delete scheduled backup shipping for a resource.

Example output could look like this:

The LINSTOR resource object is an exception to the general syntax for setting DRBD options for LINSTOR objects. With the LINSTOR resource object, you can use the drbd-peer-options to set DRBD options at the connection level between the two nodes that you specify. Specifying drbd-peer-options for a LINSTOR resource object between two nodes is equivalent to using the`linstor resource-connection drbd-peer-options` for a resource between two nodes.

For example, to set the DRBD maximum buffer size to 8192 at a connection level, for a resource named backups, between two nodes, node-0 and node-1, enter:

The command above is equivalent to the following:

Indeed, when using the linstor --curl command to examine the two commands actions on the LINSTOR REST API, the output is identical:

The connection section of the LINSTOR-generated resource file backups.res on node-0 will look something like this:

You can use the drbd-peer-options argument to set DRBD options at a connection level, between two nodes, for example:

The preceding command would set the DRBD ping-timeout option to 29.9 seconds at a connection level between two nodes, node-0 and node-1.

You can verify a LINSTOR object’s set properties by using the list-properties command, for example:

To remove a previously set DRBD option, prefix the option name with unset-. For example:

The same syntax applies to any drbd-peer-options set either on a LINSTOR resource, resource connection, or node connection. For example:

LINSTOR can convert resources between diskless and having a disk. This is achieved with the resource toggle-disk command, which has syntax similar to resource create.

For instance, add a disk to the diskless resource backups on ‘alpha’:

Remove this disk again:

To move a resource between nodes without reducing redundancy at any point, LINSTOR’s disk migrate feature can be used. First create a diskless resource on the target node, and then add a disk using the --migrate-from option. This will wait until the data has been synced to the new disk and then remove the source disk.

For example, to migrate a resource backups from ‘alpha’ to ‘bravo’:

LINSTOR expects DRBD Proxy to be running on the nodes which are involved in the relevant connections. It does not currently support connections through DRBD Proxy on a separate node.

Suppose our cluster consists of nodes ‘alpha’ and ‘bravo’ in a local network and ‘charlie’ at a remote site, with a resource definition named backups deployed to each of the nodes. Then DRBD Proxy can be enabled for the connections to ‘charlie’ as follows:

The DRBD Proxy configuration can be tailored with commands such as:

LINSTOR does not automatically optimize the DRBD configuration for long-distance replication, so you will probably want to set some configuration options such as the protocol:

LINSTOR can also be configured to automatically enable the above mentioned Proxy connection between two nodes. For this automation, LINSTOR first needs to know on which site each node is.

As the Site property might also be used for other site-based decisions in future features, the DrbdProxy/AutoEnable also has to be set to true:

A sample PostgreSQL linstor.toml looks like this:

A sample MariaDB linstor.toml looks like this:

etcd is a distributed key-value store that makes it easy to keep your LINSTOR database distributed in a HA-setup. The etcd driver is already included in the LINSTOR-controller package and only needs to be configured in the linstor.toml.

More information about how to install and configure etcd can be found here: etcd docs

And here is a sample [db] section from the linstor.toml:

To make LINSTOR’s administrative tasks more accessible and also available for web-frontends a REST API has been created. The REST API is embedded in the linstor-controller and since LINSTOR 0.9.13 configured through the linstor.toml configuration file.

The HTTP REST API can also run secured by HTTPS and is highly recommended if you use any features that require authorization. To do so you have to create a Java keystore file with a valid certificate that will be used to encrypt all HTTPS traffic.

Here is a simple example on how you can create a self signed certificate with the keytool that is included in the Java Runtime:

keytool will ask for a password to secure the generated keystore file and is needed for the LINSTOR Controller configuration. In your linstor.toml file you have to add the following section:

Client access can be restricted by using a SSL/TLS truststore on the Controller. Basically you create a certificate for your client and add it to your truststore and the client then uses this certificate for authentication.

First create a client certificate:

Then we import this certificate to our controller truststore:

And enable the truststore in the linstor.toml configuration file:

Now restart the Controller and it will no longer be possible to access the controller API without a correct certificate.

The LINSTOR client needs the certificate in PEM format, so before we can use it we have to convert the java keystore certificate to the PEM format.

To avoid entering the PEM file password all the time it might be convenient to remove the password.

Now this PEM file can easily be used in the client:

It is possible to have the LINSTOR use SSL/TLS secure TCP connection between controller and satellite connections. Without going into further details on how Java’s SSL/TLS engine works we will give you command line snippets using the keytool from Java’s runtime environment on how to configure a three node setup using secure connections.

The node setup looks like this:

Node alpha is the just the controller. Node bravo and node charlie are just satellites.

Here are the commands to generate such a keystore setup, values should of course be edited for your environment.

You can configure LINSTOR to use LDAP authentication to limit access to the LINSTOR Controller. This feature is disabled by default but you can enable and configure it by editing the LINSTOR configuration TOML file. After editing the configuration file, you will need to restart the linstor-controller.service. An example LDAP section within the configuration file looks like this:

After configuring the LINSTOR Controller to authenticate users through LDAP (or LDAPS), and the LINSTOR REST API HTTPS, you will need to enter LINSTOR commands as follows:

If you have configured LDAP authentication without also configuring LINSTOR REST API HTTPS, you will need to explicitly enable password authentication over HTTP, by using the --allow-insecure-path flag with your linstor commands. This is not recommended outside of a secured and isolated LAN, as you will be sending credentials in plain text.

LINSTOR automatically configures quorum policies on resources when quorum is achievable. This means, whenever you have at least two diskful and one or more diskless resource assignments, or three or more diskful resource assignments, LINSTOR will enable quorum policies for your resources automatically.

Inversely, LINSTOR will automatically disable quorum policies whenever there are less than the minimum required resource assignments to achieve quorum.

This is controlled through the, DrbdOptions/auto-quorum, property which can be applied to the linstor-controller, resource-group, and resource-definition. Accepted values for the DrbdOptions/auto-quorum property are disabled, suspend-io, and io-error.

Setting the DrbdOptions/auto-quorum property to disabled will allow you to manually, or more granularly, control the quorum policies of your resources should you want to.

For example, to manually set the quorum policies of a resource-group named my_ssd_group, you would use the following commands:

You may want to disable DRBD’s quorum features completely. To do that, you would need to first disable DrbdOptions/auto-quorum on the appropriate LINSTOR object, and then set the DRBD quorum features accordingly. For example, use the following commands to disable quorum entirely on the my_ssd_group resource-group:

If a satellite is offline for a prolonged period of time, LINSTOR can be configured to declare that node as evicted. This triggers an automated reassignment of the affected DRBD resources to other nodes to ensure a minimum replica count is kept.

This feature uses the following properties to adapt the behaviour.

After the linstor-controller loses the connection to a satellite, aside from trying to reconnect, it starts a timer for that satellite. As soon as that timer exceeds DrbdOptions/AutoEvictAfterTime and all of the DRBD-connections to the DRBD-resources on that satellite are broken, the controller will check whether or not DrbdOptions/AutoEvictMaxDisconnectedNodes has been met. If it hasn’t, and DrbdOptions/AutoEvictAllowEviction is true for the node in question, the satellite will be marked as EVICTED. At the same time, the controller will check for every DRBD-resource whether the number of resources is still above DrbdOptions/AutoEvictMinReplicaCount. If it is, the resource in question will be marked as DELETED. If it isn’t, an auto-place with the settings from the corresponding resource-group will be started. Should the auto-place fail, the controller will try again later when changes that might allow a different result, such as adding a new node, have happened. Resources where an auto-place is necessary will only be marked as DELETED if the corresponding auto-place was successful.

The evicted satellite itself will not be able to reestablish connection with the controller. Even if the node is up and running, a manual reconnect will fail. It is also not possible to delete the satellite, even if it is working as it should be. The satellite can, however, be restored. This will remove the EVICTED-flag from the satellite and allow you to use it again. Previously configured network interfaces, storage pools, properties and similar entities as well as non-DRBD-related resources and resources that could not be autoplaced somewhere else will still be on the satellite. To restore a satellite, use

By setting the DrbdOptions/auto-diskful option on a LINSTOR resource definition , you are configuring the LINSTOR controller to make a Diskless DRBD resource Diskful if the resource has been in a DRBD Primary state for more than the specified number of minutes. After the specified number of minutes, LINSTOR will automatically use the resource toggle-disk command to toggle the resource state on the Diskless node, for the given resource.

To set this property, for example, on a LINSTOR resource definition named myres with a threshold of five minutes, enter the command:

To unset the LINSTOR auto-diskful option, enter:

These LINSTOR properties may be set using the set-property command and may be set on the following objects: volume, storage pool, resource, controller, or node. You can also set these QoS properties on resource groups, volume groups, resource definitions, or volume definitions. When you set a QoS property on a group or definition, resources created from the group or definition will inherit the QoS settings.

The following example shows creating a resource group, then creating a volume group, then applying QoS settings to the volume group, and then spawning resources from the resource group. A verification command will show that the spawned resources inherit the QoS settings. The example uses an assumed previously created LINSTOR storage pool named pool1. You will need to replace this name with a storage pool name that exists in your environment.

To verify that the spawned resources inherited the QoS setting, you can show the contents of the corresponding sysfs file, on a node that contributes storage to the storage pool.

A single LINSTOR volume can be composed of multiple DRBD devices. For example, DRBD with external metadata will have three backing devices: a data (storage) device, a metadata device, and the composite DRBD device (volume) provided to LINSTOR. If the data and metadata devices correspond to different backing disks, then if you set a sysfs property for such a LINSTOR volume, only the local data (storage) backing device will receive the property value in the corresponding /sys/fs/cgroup/blkio/ file. Neither the device backing DRBD’s metadata, nor the composite backing device provided to LINSTOR would receive the value. However, when DRBD’s data and its metadata share the same backing disk, QoS settings will affect the performance of both data and metadata operations.

In case a LINSTOR resource definition has an nvme-target as well as an nvme-initiator resource, both data (storage) backing devices of each node will receive the sysfs property value. In case of the target, the data backing device will be the volume of either LVM or ZFS, whereas in case of the initiator, the data backing device will be the connected nvme-device, regardless of which other LINSTOR layers, such as LUKS, NVMe, DRBD, and others (see Using LINSTOR Without DRBD), are above that.

在命令行中列出可用命令的一种快速方法是键入 linstor 。

Further information about subcommands (e.g., list-nodes) can be retrieved in two ways:

当LINSTOR以交互模式（LINSTOR interactive）执行时，使用 help 子命令尤其有用。

LINSTOR最有用的特性之一是它丰富的tab-completion，它基本上可以用来完成LINSTOR所知道的每个对象（例如，节点名、IP地址、资源名等等）。在下面的示例中，我们将展示一些可能的完成及其结果：

如果制表符完成不正常，请尝试获取相应的文件：

For zsh shell users, the linstor-client command can generate a zsh compilation file, that has basic support for command and argument completion.

If something goes wrong and you need help finding the cause of the issue, you can use

The command above will create a new sos-report in /var/log/linstor/controller/ on the controller node. Alternatively you can use

which will create a new sos-report and additionally downloads that report to the local machine into your current working directory.

This sos-report contains logs and useful debug-information from several sources (Linstor-logs, dmesg, versions of external tools used by LINSTOR, ip a, database dump and many more). These information are stored for each node in plain text in the resulting .tar.gz file.

如需社区帮助，请订阅我们的邮件列表： https://lists.linbit.com/listinfo/drbd-user

要提交bug或功能请求，请查看我们的GitHub页面 https://GitHub.com/linbit

Alternatively, if you need to purchase remote installation services, 24/7 support, access to certified repositories, or feature development, please contact us: +1-877-454-6248 (1-877-4LINBIT) , International: +43-1-8178292-0 | [email protected]

LINSTOR Operator v2 is deployed using kustomize tool, integrated with kubectl.

The Operator v1 is installed using a Helm v3 chart as follows:

The LINSTOR Operator can automate some basic storage set up for LINSTOR.

This section describes the different options for enabling security features available when using this operator. The following guides assume the operator is installed using Helm

The Helm charts provide a set of further customization options for advanced use cases.

You can use Prometheus to monitor LINSTOR components. The operator will set up monitoring containers along the existing components and make them available as a Service.

If you use the Prometheus Operator, the LINSTOR Operator will also set up the ServiceMonitor instances. The metrics will automatically be collected by the Prometheus instance associated to the operator, assuming watching the Piraeus namespace is enabled.

To disable exporting of metrics, set operator.satelliteSet.monitoringImage to an empty value.

The operator can configure the satellites and CSI plug-in to use an existing LINSTOR setup. This can be useful in cases where the storage infrastructure is separate from the Kubernetes cluster. Volumes can be provisioned in diskless mode on the Kubernetes nodes while the storage nodes will provide the backing disk storage.

To skip the creation of a LINSTOR Controller deployment and configure the other components to use your existing LINSTOR Controller, use the following options when running helm install:

After all pods are ready, you should see the Kubernetes cluster nodes as satellites in your LINSTOR setup.

Create a storage class referencing an existing storage pool on your storage nodes.

You can provision new volumes by creating PVCs using your storage class. The volumes will first be placed only on nodes with the given storage pool, that is, your storage infrastructure. Once you want to use the volume in a pod, LINSTOR CSI will create a diskless resource on the Kubernetes node and attach over the network to the diskful resource.

在Kubernetes中由社区支持的LINSTOR部署版本称为Piraeus。Piraeus项目提供了一个 an operator 进行部署。

The following storage class contains all currently available parameters to configure the provisioned storage.

The csi.storage.k8s.io/fstype parameter sets the file system type to create for volumeMode: FileSystem PVCs. Currently supported are:

autoPlace is an integer that determines the amount of replicas a volume of this StorageClass will have. For instance, autoPlace: "3" will produce volumes with three-way replication. If neither autoPlace nor nodeList are set, volumes will be automatically placed on one node.

placementCount is an alias for autoPlace

The LINSTOR Resource Group (RG) to associate with this StorageClass. If not set, a new RG will be created for each new PVC.

storagePool 是用于为新创建的卷提供存储的LINSTOR storage pool的名称。

disklessStoragePool 是一个可选参数，它只影响作为客户端无磁盘分配给 kubelets 的LINSTOR卷。如果在LINSTOR中定义了自定义 diskless storage pool，请在此处指定。

A comma-separated list of layers to use for the created volumes. The available layers and their order are described towards the end of this section. Defaults to drbd,storage

Select from one of the available volume schedulers:

Control on which nodes a volume is accessible. The value for this option can take two different forms:

encryption is an optional parameter that determines whether to encrypt volumes. LINSTOR must be configured for encryption for this to work properly.

nodeList 是要分配给卷的节点列表。这将把卷分配给每个节点，并在所有节点之间进行复制。这也可以用于按主机名选择单个节点，但使用replicasOnSame选择单个节点更灵活。

clientList is a list of nodes for diskless volumes to be assigned to. Use in conjunction with 节点列表.

replicasOnSame is a list of key or key=value items used as autoplacement selection labels when autoplace is used to determine where to provision storage. These labels correspond to LINSTOR node properties.

Let’s explore this behavior with examples assuming a LINSTOR cluster such that node-a is configured with the following auxiliary property zone=z1 and role=backups, while node-b is configured with only zone=z1.

如果我们使用 autoPlace: "1" ` 和 `replicasOnSame: "zone=z1 role=backups"`来配置一个 StorageClass ，那么从该 StorageClass 创建的所有卷都将配置在 node-a 上，因为这是LINSTOR集群中唯一具有所有正确的key=value对的节点。这是选择单个节点进行资源调配的最灵活方式。

如果我们使用 autoPlace: "1" ` 和 `replicasOnSame: "zone=z1" ` 来配置一个 StorageClass ，那么卷将在 `node-a 或 node-b 上配置，因为它们都有 zone=z1 辅助属性。

If we configure a StorageClass with autoPlace: "2" and replicasOnSame: "zone=z1 role=backups", then provisioning will fail, as there are not two or more nodes that have the appropriate auxiliary properties.

如果我们用 autoPlace: "2" 和 replicasOnSame: "zone=z1" 配置一个存储类，那么卷将同时在 node-a 和 node-b 上配置，因为它们都有 zone=z1 辅助属性。

You can also use a property key without providing a value to ensure all replicas are placed on nodes with the same property value, with caring about the particular value. Assuming there are 4 nodes, node-a1 and node-a2 are configured with zone=a. node-b1 and node-b2 are configured with zone=b. Using autoPlace: "2" and replicasOnSame: "zone" will place on either node-a1 and node-a2 OR on node-b1 and node-b2.

replicasOnDifferent takes a list of properties to consider, same as replicasOnSame. There are two modes of using replicasOnDifferent:

Create a diskless resource on all nodes that were not assigned a diskful resource.

Do not place the resource on a node which has a resource with a name matching the regex.

fsOpts 是一个可选参数，在创建时将选项传递给卷的文件系统。

mountOpts 是一个可选参数，在装载时将选项传递给卷的文件系统。

Extra arguments to pass to xfs_io, which gets called before right before first use of the volume.

Parameters starting with property.linstor.csi.linbit.com/ are translated to LINSTOR properties that are set on the Resource Group associated with the StorageClass.

For example, to set DrbdOptions/auto-quorum to disabled, use:

The full list of options is available here

Advanced DRBD options to pass to LINSTOR. For example, to change the replication protocol, use DrbdOptions/Net/protocol: "A".

LINSTOR supports the volume snapshot feature, which is configured in some, but not all Kubernetes distributions.

To check if your Kubernetes distribution supports snapshots out of the box, run the following command:

In case your Kubernetes distribution does not support snapshots, you can manually add the required components from the Kubernetes Storage SIG. For convenience, you can use Helm Charts provided by the Piraeus team to add these components.

Then we can create our VolumeSnapshotClass:

使用 kubectl 创建 VolumeSnapshotClass ：

现在，我们将为上面创建的卷创建卷快照。这是用 VolumeSnapshot 完成的：

使用 kubectl 创建 VolumeSnapshot ：

You can check that the snapshot creation was successful

最后，我们将使用 PVC 从快照创建一个新卷。

用 kubectl 创建 PVC ：

The following example show common scenarios where you want to optimize volume accessibility and locality. It also includes examples of how to spread volume replicas across zones in a cluster.

The LINSTOR Operator will create a backup of the database before upgrading. This makes it possible to roll back to a known state should something go wrong during the upgrade.

There are situations in which the Operator can’t create the backup automatically. This might be because:

Some versions require special steps, see below.

To create a backup of the etcd database and store it on your control host, run:

These commands will create a file save.db on the machine you are running kubectl from.

The next steps are exactly the same as in Kubernetes:

The LINSTOR container images come with the LINBIT GUI preinstalled. To expose it on your cluster, configure an OpenShift Route resource:

The GUI is now accessible at https://linstor-op-cs-storage.apps.oc.example.com/ui/

OpenShift includes a fully configured monitoring stack. While most of the monitoring stack is only intended for Openstack infrastructure, basic monitoring for LINBIT SDS is possible.

First, ensure that monitoring of user-defined projects is enabled in OpenShift by following the steps in Red Hat documentation.

Once monitoring for user-defined projects is enabled, the LINSTOR Operator automatically detects the presence of OpenShift’s Prometheus-based monitoring stack and configures the ServiceMonitor resources. The Prometheus instance will scrape metrics for DRBD and LINSTOR from all Cluster nodes.

To run LINSTOR, every Nomad agent needs to be configured to:

The LINSTOR Controller is deployed as a service with no replicas. At any point in time, there can only be one LINSTOR Controller running in a cluster. It is possible to restart the controller on a new node, provided it still has access to the database. See Creating a Highly Available LINSTOR Cluster for more information.

The following example will create a Nomad job starting a single LINSTOR Controller in datacenter dc1 and connect to an external database.

Apply the job by running:

In Nomad, the LINSTOR satellites are deployed as a system job that runs in a privileged container. In addition to the satellites, the job will also load the DRBD module along with other kernel modules used by LINSTOR.

The following example will create a Nomad job starting a LINSTOR satellite on every node in data center dc1.

Apply the job by running:

Once the linstor-controller and linstor-satellite jobs are running, you can start configuring the cluster using the linstor command line tool.

This can done:

In all cases, you need to add the satellites to your cluster and create some storage pools. For example, to add the node nomad-01.example.com and configure a LVM Thin storage pool, you would run:

To use the volume in a job, add the volume and volume_mount stanzas to the job specification:

LINSTOR can create snapshots of existing volumes, provided the underlying storage pool driver supports snapshots.

The following command creates a snapshot named snap1 of the volume vol1.

You can use a snapshot to pre-populate an existing volume with data from the snapshot

Version 7 of the plugin uses a LINSTOR controller API that is available from LINSTOR version 1.21.1 onwards. Make sure that your LINSTOR setup (controller and satellites) use at least that version.

Version 5 of the plug-in drops compatibility with the legacy configuration options “storagepool” and “redundancy”. Version 5 requires a “resourcegroup” option, and obviously a LINSTOR resource group. The old options should be removed from the configuration.

Configuring LINSTOR is described in Section Configuring LINSTOR, a typical example follows:

Let’s assume the pool was set to “mypool”, and redundancy to 3.

Version 6.0.0 of the plug-in drops all code related to the redundancy setting. This is handled by LINSTOR resource groups (resourcegroup setting) for a very long time. No change should be required.

The controllervm setting, which was intended for executing a LINSTOR controller in a VM manged by LINSTOR is gone. Using drbd-reactor to realize a highly available LINSTOR controller is what we suggest.

The settings statuscache and preferlocal are now enabled by default.

With version 6 PVE added additional parameters to some functions and rightfully reset their “APIAGE”. This means that old plug-ins, while actually usable as they don’t use any of these changed functions do not work anymore. Please upgrade to plug-in version 5.2.1 at least.

To use LINSTOR in Proxmox, you will need to install the DRBD kernel module. The DRBD 9 kernel module is installed as a kernel module source package (drbd-dkms). Therefore, you will have to install the Proxmox VE kernel headers package, pve-headers, before you install the DRBD kernel module from LINBIT’s repositories. Following that order ensures that the kernel module will build properly for your kernel.

If you do not plan to install the latest Proxmox kernel, you have to install kernel headers matching your current running kernel (for example, pve-headers-$(uname -r)). If you missed this step, then you can still rebuild the drbd-dkms package against your current kernel (so long as you have installed kernel headers in advance) by entering the apt install --reinstall drbd-dkms command.

If you are a LINBIT customer, or you have an evaluation account, you can enable LINBIT’s drbd-9 repository on your node and then update your repositories by using an apt update command.

You can then install the DRBD kernel module, DRBD utilities, and the LINSTOR Proxmox plug-in by entering:

LINBIT provides a dedicated public repository for Proxmox VE users. This repository not only contains the Proxmox plug-in, but the whole DRBD SDS stack including a DRBD SDS kernel module and user-space utilities.

LINBIT’s repository can be enabled as follows, where “$PVERS” should be set to your Proxmox VE major version (for example, “7”, not “7.1”):

修改 /etc/one/oned.conf 的以下部分`

将linstor添加到 TM MAD 和 DATASTORE MAD 部分的驱动程序列表中：

添加新的TM_MAD_CONF和DS_MAD_CONF部分：

After making these changes, restart the OpenNebula service.

The Front-End node issues commands to the Storage and Host nodes through LINSTOR.

存储节点在本地保存vm的磁盘映像。

Host nodes are responsible for running instantiated VMs and typically have the storage for the images they need attached across the network through LINSTOR diskless mode.

All nodes must have DRBD9 and LINSTOR installed. This process is detailed in the User’s Guide for DRBD9

It is possible to have Front-End and Host nodes act as storage nodes in addition to their primary role provided that they the meet all the requirements for both roles.

The oneadmin, “Cloud Administrator”, user account must have passwordless sudo access to the mkfs command on the Storage nodes

Create a datastore configuration file named ds.conf and use the onedatastore tool to create a new datastore based on that configuration. There are two mutually exclusive deployment options: LINSTOR_AUTO_PLACE and LINSTOR_DEPLOYMENT_NODES. If both are configured, LINSTOR_AUTO_PLACE is ignored. For both of these options, BRIDGE_LIST must be a space separated list of all storage nodes in the LINSTOR cluster.

因为1.0.0版LINSTOR支持资源组。资源组是所有链接到该资源组的资源共享的设置的集中点。

Create a resource group and volume group for your datastore, it is mandatory to specify a storage-pool within the resource group, otherwise monitoring space for OpenNebula will not work. Here we create one with 2 node redundancy and use a created opennebula-storagepool:

Now add a OpenNebula datastore using the LINSTOR plug-in:

The following attributes are deprecated and were removed in version 2.0.

LINSTOR driver can also be used as a system datastore, configuration is pretty similar to normal datastores, with a few changes:

还要将新的sys datastore id添加到镜像数据存储的 COMPATIBLE_SYS_DS （以逗号分隔），否则调度程序将忽略它们。

如果要使用易失性磁盘进行实时迁移，则需要为KVM启用 --unsafe 选项，请参见： opennebula doc

If this is a fresh installation, skip this section and continue with Installing the LINSTOR Driver.

Starting with OpenStack Stein, the LINSTOR driver is part of the Cinder project. While the driver can be used as is, it might be missing features or fixes available in newer version. Due to OpenStacks update policy for stable versions, most improvements to the driver will not get back-ported to older stable releases.

LINBIT maintains a fork of the Cinder repository with all improvements to the LINSTOR driver backported to the supported stable versions. Currently, these are:

The exact steps to enable the LINSTOR Driver depend on your OpenStack distribution. In general, the python-linstor package needs to be installed on all hosts running the Cinder volume service. The next section will cover the installation process for common OpenStack distributions.

The Transport Protocol in Cinder is how clients (for example nova-compute) access the actual volumes. With LINSTOR, you can choose between two different drivers that use different transports.

To verify that all back ends are up and running, you can use the OpenStack command line client:

If you have the Horizon GUI deployed, check Admin > System Information > Block Storage Service instead.

In the above example all configured services are enabled and up. If there are any issues, please check the logs of the Cinder Volume service.

Each volume type can be customized by adding properties or “Extra Specs” as they are called in the Horizon GUI.

To add a property to a volume type on the command line use:

Alternatively, you can set the property using the GUI by navigating tp Admin > Volume > Volume Types. In the Actions column, open the dropdown menu and click the View Extra Specs button. This opens a dialog you can use to create, edit and delete properties.

Every volume and snapshot has a Messages tab in the Horizon dashboard. In case of errors, the list of messages can be used as a starting point for further investigation. Some common messages in case of errors:

There was an error creating a new volume. Check the Cinder Volume service logs for more details

If this is the only error message, this means the Cinder Scheduler could not find a volume back end suitable for creating the volume. This is most likely because:

The LINSTOR driver is called as part of the Cinder Volume service.

Some issues will not be logged in the Cinder Service but in the actual consumer of the volumes, most likely the compute service (Nova). As with the volume service, the exact host and location to check depends on your OpenStack distribution: