Understanding LINSTOR Gateway

The server role is LINSTOR Gateway’s main mode of operation. The main task of the server component is to communicate with the LINSTOR controller.

This is any piece of software that interacts with LINSTOR Gateway via LINSTOR Gateway’s REST API. In most cases, this will be the command line client that is included with LINSTOR Gateway.

It is the interface between LINSTOR Gateway and the user.

Even if there are multiple LINSTOR Gateway servers in a cluster, the client only ever talks to one of them.

The agent is a more abstract role for a node, in the sense that it does not necessarily run any part of the LINSTOR Gateway software itself. Instead, it contains the software components that LINSTOR Gateway uses to provide its services, for example:

To be a proper part of the cluster, an agent node also requires the essential components of a LINBIT SDS software stack, such as:

The health check integrated into the LINSTOR Gateway Client binary can be used to identify which components are still missing or misconfigured on a given node.

iSCSI is a transport protocol that allows SCSI traffic to be sent via TCP. The standard has seen wide use since its inception in the early 2000s, so it has often been viewed as the “default choice” for network-attached storage.

LINSTOR Gateway uses an iSCSI target implementation that is included in the Linux kernel, LIO.

NVMe over Fabrics is a much newer standard relative to iSCSI. It allows routing NVMe traffic over several different physical transports, such as RDMA, Ethernet (TCP/IP) or Fibre Channel.

Linux kernel support for NVMe over Fabrics targets is — generally speaking — more actively maintained than support for iSCSI target implementations. Using NVMe-oF might lead to throughput improvements in a storage cluster, especially when using modern high-performance hardware.

LINSTOR Gateway uses the NVMe target implementation bundled with the Linux kernel.

The Network File System (NFS) serves a different purpose than iSCSI and NVMe-oF.

Rather than transmitting block-level data over the network, NFS is a distributed file system. NFS exports are often used to share directories across a network. One common use case would be providing images of operating system installation media to virtualization hosts.

LINSTOR Gateway uses the NFS server implementation that is included with Linux.

Here are two guidelines for all LINSTOR Gateway deployments:

Some example real-world deployment scenarios follow to show how these rules apply.

This is the simplest possible setup for a LINSTOR cluster. There is one node that is chosen to be the LINSTOR controller node, and the LINSTOR controller service can only run on this particular node.

In this case, the rules from above boil down to one trivial instruction:

You can omit the second rule because the only possible node that the LINSTOR controller can run on is node-1. From the perspective of node-1, this is equivalent to localhost. As it so happens, the default place the LINSTOR Gateway server searches for the LINSTOR controller is localhost, so the default configuration is sufficient here.

This setup is very simple to implement, but it has an important drawback: the LINSTOR controller on node-1 is the single point of failure for the cluster’s control plane. If node-1 were to go offline, you would lose the ability to control the storage in your cluster.

To address this issue and make the cluster more robust against node failure, a highly available LINSTOR controller can be configured.

Here, the picture is different: The LINSTOR controller is currently running on node-1. However, if node-1 should fail, one of the other nodes can take over. This configuration is more complex, but it makes sense in environments where it is critical that the LINSTOR controller stays available at all times.

In a cluster such as this, the general rules outlined above resolve to these instructions:

In order for LINSTOR Gateway to be able to export highly available storage, it must be able to create said highly available storage in the first place.

To do this, it relies on the bottom-most part of the LINBIT software stack: DRBD, a “tried and true” solution for data replication and high availability clusters.

If you are not yet familiar with DRBD and want to learn more, you can refer to its User’s Guide. A deep understanding of DRBD is not required to use LINSTOR Gateway effectively, but it helps – especially when troubleshooting should any issues arise.

For a shortened – but also massively oversimplified – explanation, you can think of DRBD as a sort of “software RAID-1 over the network”.

Highly available storage via DRBD is great, but it can be difficult to manage and manipulate resources at scale. To handle this – and many other orchestration tasks – LINSTOR was created.

As the name might imply, LINSTOR Gateway heavily relies on LINSTOR to create the DRBD resources that can later be exported as highly available storage.

Knowledge and experience with handling LINSTOR clusters helps immensely with using and understanding LINSTOR Gateway, as most of the required administration tasks are carried out within LINSTOR.

The piece of software that enables LINSTOR Gateway to actually export the LINSTOR resources it created as highly available storage is DRBD Reactor, more specifically its promoter plugin.

DRBD Reactor – again, as the name implies – reacts to DRBD events. DRBD already includes sophisticated state management mechanisms to ensure that only one node in the cluster has write access to a resource at a time. In the promoter plugin, we use this to our advantage to make sure that a particular service is always started on the same node that is allowed to write to the resource.

This might remind you of other cluster managers, such as Pacemaker. In fact, DRBD Reactor was inspired by such cluster managers and its goals and features overlap with some of Pacemaker’s.

However, administrators often face difficulties when implementing highly available clusters using these cluster managers in combination with DRBD, due to the sheer complexity of such a system.

This is why the DRBD Reactor promoter plugin was intentionally designed as a very simple cluster manager that is useful with minimal configuration needed for existing DRBD deployments.

The DRBD Reactor promoter plugin needs a service to keep highly available. Fortunately, it was built with cross-compatibility to Pacemaker in mind. Because of this, Pacemaker’s OCF Resource Agents are supported in DRBD Reactor.

This is very convenient because it allows LINSTOR Gateway to re-use existing code that makes services such as iSCSI targets and NFS exports highly available.

Finally, this is where LINSTOR Gateway comes in.

In a nutshell, LINSTOR Gateway actually only does two things:

Of course, there are still challenges associated with “doing it right” for every possible use-case and context that highly available storage might be used in.

This is why LINSTOR Gateway aims to automate away as much of the setup and maintenance work as possible, so that the administrator can focus on using their highly available storage effectively.