LINSTOR®, developed by LINBIT®, is a robust open source software-defined storage system designed for managing block storage devices in large-scale clusters. It simplifies the creation, management, and replication of storage resources across diverse environments, enhancing both performance and reliability. Securing LINSTOR is crucial to protect sensitive data, ensure system integrity, and maintain high availability, particularly in production and cloud deployments.
LINSTOR has been developed with security options that when used can help organizations safeguard their storage solution. While some of these details can be found throughout the LINSTOR User’s Guide, this blog will serve as an aggregate for the best practices for securing a LINSTOR cluster.
Portions of this blog will demonstrate the creation and distribution of cryptographic keys. Maintaining a Public Key Infrastructure (PKI) on an offline system is recommended to protect sensitive cryptographic keys from potential cyber threats and unauthorized access. This isolation significantly reduces the risk of key compromise, ensuring the integrity and trustworthiness of the PKI. All command examples in this blog that generate private keys and certificates will be run on the “PKI server”, and the resulting keys will be distributed to the necessary nodes only.
The example cluster in this blog is made up of four nodes. One LINSTOR controller node named linstor-ctrl-0, and three LINSTOR satellite nodes named linstor-sat-0, linstor-sat-1, and linstor-sat-2. The following command entered on the PKI server will create the directory structure necessary for following the examples in this blog:
# for n in ctrl-0 sat-0 sat-1 sat-2; do
mkdir linstor-$n
done
mkdir linstor-ctrl-0/ssl
Securing the LINSTOR Controller’s REST API
The LINSTOR controller service has a REST API listening on TCP port 3370 of all network interfaces of the LINSTOR controller node by default.
You can verify this using a simple curl command from any computer with network access to the controller.
curl -X 'GET' 'http://192.168.222.250:3370/v1/controller/version' -H 'accept: application/json'
📝 NOTE: The LINSTOR controller in the examples throughout this blog will have the IP address, 192.168.222.250.
Configuring Listening Address and Port Number
The first and easiest thing an administrator can do increase the security of the LINSTOR controller’s REST API endpoint, is configure which interface the endpoint listens on. This is done by adding or editing the LINSTOR controller’s configuration TOML found in /etc/linstor/linstor.toml. The HTTP endpoint can be configured with the following settings:
[http]
enabled = true
port = 3370
listen_addr = "192.168.222.250" # "127.0.0.1" disables remote access
Configure the listen_addr setting to an interface that is on a trusted network that only authorized LINSTOR clients would have access to. In some cases you can disable remote access completely, limiting access to LINSTOR’s REST API to the host running the LINSTOR controller service by configuring listen_addr = "127.0.0.1".
💡 TIP: In highly available LINSTOR controller deployments, you would need to configure a virtual IP (VIP) address as the listen_addr and the address that LINSTOR clients use to access the LINSTOR controller.
Securing LINSTOR Client Access With HTTPS
By default, LINSTOR clients will connect to the LINSTOR controller’s REST API via HTTP. HTTP is a plain text protocol and therefore any information communicated over it can be intercepted, read, and even altered in transit as in a man-in-the-middle attack.
The screen grab below shows the information returned by the LINSTOR controller when a linstor node list command was run from a LINSTOR client on the network using HTTP:
To use HTTPS, the TLS secured HTTP transport, for LINSTOR’s REST API you’ll first need to generate a certificate for the LINSTOR controller. Run the following command on the PKI server to generate the LINSTOR controller’s certificate stored in a Java KeyStore (JKS):
# keytool -keyalg rsa -keysize 2048 -genkey -keystore linstor-ctrl-0/keystore.jks \
-storepass linstor -keypass linstor -alias linstor-ctrl-0 \
-dname "CN=linstor-ctrl-0, OU=controller, O=LINBIT, L=Vienna, ST=Austria, C=AT"
❗ IMPORTANT: When choosing the keysize and key algorithm used for your own cluster, you should consult and adhere to the security policies of your own organization.
Securely copy the generated JKS to the controller node placing it in the /etc/linstor/ directory.
With the keystore.jks in place on the controller, open the LINSTOR controller service’s configuration file, /etc/linstor/linstor.toml, in an editor and append the following lines to configure the HTTPS endpoint for the controller’s REST API:
[https]
enabled = true
port = 3371
listen_addr = "192.168.222.250"
keystore = "/etc/linstor/keystore.jks"
keystore_password = "linstor"
To load the new settings, restart the LINSTOR controller service by using systemctl:
# systemctl restart linstor-controller.service
Now, the LINSTOR controller should be listening on an additional port, TCP 3371, where HTTPS is used for handling client requests instead of HTTP. To verify this you can test the endpoint:
# curl -k -X 'GET' 'https://192.168.222.250:3371/v1/controller/version' -H 'accept: application/json'
Inspecting another packet capture after enabling HTTPS shows that SSL/TLS is now encrypting the traffic between the LINSTOR Controller and LINSTOR Client:
If you don’t disable the HTTP endpoint on the LINSTOR controller, LINSTOR clients attempting a connection over HTTP will be redirected to the HTTPS endpoint by a HTTP 302 response code.
# curl -iLk -X 'GET' 'http://192.168.222.250:3370/v1/controller/version' -H 'accept: application/json'
HTTP/1.1 302 Found
Location: https://192.168.222.250:3371/v1/controller/version
Content-Language: en-US
Content-Type: text/html;charset=ISO-8859-1
Transfer-Encoding: chunked
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: origin, content-type, accept, authorization
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS, HEAD
Content-Type: application/json
Content-Length: 143
{"version":"1.27.1","git_hash":"c6f8ceed9d50da2c4d37ae8ce20d09daf3046464","build_time":"2024-04-25T11:12:13+00:00","rest_api_version":"1.22.0"}
To avoid the redirect, you can update the LINSTOR client configuration as shown in the following modified configuration file:
# cat /etc/linstor/linstor-client.conf
[global]
controllers=linstor+ssl://192.168.222.250:3371
Once you’re certain that all LINSTOR clients are configured to connect to the LINSTOR controller by using the HTTPS endpoint on port 3371, you should completely disable the HTTP endpoint on the LINSTOR controller. You can do this by adding the following section and key value pair to the /etc/linstor/linstor.toml configuration file.
[http]
enabled = false
Restricting Client Access by Using SSL/TLS Certificates
You can restrict access to the LINSTOR controller service by using trusted certificates. The LINSTOR controller will need to import the certificates of any node that will make LINSTOR client requests into a Java TrustStore that the LINSTOR controller will use to authenticate clients that attempt to connect to it.
In most clusters, the LINSTOR controller and LINSTOR satellite nodes will be configured with a LINSTOR client to manage the LINSTOR cluster. That is the model the example cluster follows in this blog. However, it is possible to have a non-controller and non-satellite system with a LINSTOR client configured to manage your cluster. The only constraint would be that such a client would need to have a network route to the LINSTOR controller node. To make this possible, you would create a certificate for the LINSTOR client system and import it into the LINSTOR controller’s truststore, as is shown for the controller and satellite systems in the following examples.
Because you already created the LINSTOR controller certificate and keystore in the previous section, you only need to create the LINSTOR satellite certificates on the PKI server:
# for i in {0..2}; do
keytool -keyalg rsa -keysize 2048 -genkey -keystore linstor-sat-$i/keystore.jks \
-storepass linstor -keypass linstor -alias linstor-sat-$i \
-dname "CN=linstor-sat-$i, OU=satellite, O=LINBIT, L=Vienna, ST=Austria, C=AT"
done
Next, import the certificates for the LINSTOR controller and LINSTOR satellites into the LINSTOR controller’s truststore:
# keytool -importkeystore -srcstorepass linstor -deststorepass linstor -keypass linstor \
-srckeystore linstor-ctrl-0/keystore.jks -destkeystore linstor-ctrl-0/truststore.jks
# for i in 0 1 2; do
keytool -importkeystore -srcstorepass linstor -deststorepass linstor -keypass linstor \
-srckeystore linstor-sat-$i/keystore.jks -destkeystore linstor-ctrl-0/truststore.jks
done
Securely copy the populated linstor-ctrl-0/truststore.jks truststore to the controller node, placing it in the /etc/linstor/ directory.
Enable the truststore in the LINSTOR controller’s linstor.toml configuration file:
[https]
[ ... ]
truststore = "/etc/linstor/truststore.jks"
truststore_password = "linstor"
Finally, restart the LINSTOR controller service:
# systemctl restart linstor-controller.service
After restarting the LINSTOR controller service, you will no longer be able to access the LINSTOR controller without using a trusted certificate.
The LINSTOR client needs its certificate to be in PEM format. To convert the java keystore certificate to PEM format, use the openssl utility supplying the password used when creating the Java keystore certificate, and providing a new password to be applied to the PEM file:
# for i in linstor-ctrl-0 linstor-sat-{0..2}; do
keytool -importkeystore -srckeystore $i/keystore.jks -destkeystore $i/client.p12 \
-storepass linstor -keypass linstor -srcalias $i -srcstoretype jks -deststoretype pkcs12
done
# for i in linstor-ctrl-0 linstor-sat-{0..2}; do
openssl pkcs12 -in $i/client.p12 -out $i/client_with_pw.pem
done
When you enter commands from the LINSTOR client, you will also need to enter the password that you just applied to the PEM. You can remove the password from the client certificate for convenience by using the following commands:
# for i in linstor-ctrl-0 linstor-sat-{0..2}; do
openssl rsa -in $i/client_with_pw.pem -out $i/client.pem
done
# for i in linstor-ctrl-0 linstor-sat-{0..2}; do
openssl x509 -in $i/client_with_pw.pem >> $i/client.pem
done
Securely copy each of the generated client.pem certificates to its corresponding client system, storing it in /etc/linstor/. Add the certfile option to the LINSTOR client configuration in /etc/linstor/linstor-client.conf to configure certificate authentication to the LINSTOR controller:
[global]
controllers = linstor+ssl://192.168.222.250:3371
certfile = /etc/linstor/client.pem
After making this configuration change, you should once again be able to access the LINSTOR controller from each of the LINSTOR clients in your cluster.
Securing Satellite Connections
It is also possible to have LINSTOR use SSL/TLS on the communications between LINSTOR controllers and LINSTOR satellites. This will encrypt the traffic between the LINSTOR controller and LINSTOR satellites, safeguarding it from snooping and modification in transit.
The LINSTOR satellite nodes are technically “servers” listening for commands from clients. In this case, the LINSTOR controller is a client (not to be confused with the LINSTOR client that you use to enter commands sent to the LINSTOR controller). Because of this, you need to import the LINSTOR controller’s certificates into truststores for each LINSTOR satellite node. Also, since the examples have used self-signed certs, you will need to import the LINSTOR satellite certificate into the LINSTOR controller’s truststore.
Create the LINSTOR satellite certificate truststores first:
# for i in linstor-sat-{0..2}; do
keytool -importkeystore -srcstorepass linstor -deststorepass linstor -keypass linstor \
-srckeystore linstor-ctrl-0/keystore.jks -destkeystore $i/certificates.jks
done
Next, create the LINSTOR controller’s certificate truststore.
📝 NOTE: Here, you are creating a new truststore for the satellite certificates that is separate from the LINSTOR client trusted certificates that you configured earlier.
# for i in linstor-sat-{0..2}; do
keytool -importkeystore -srcstorepass linstor -deststorepass linstor -keypass linstor \
-srckeystore $i/keystore.jks -destkeystore linstor-ctrl-0/ssl/certificates.jks
done
❗ IMPORTANT: The LINSTOR controller’s truststore and keystore for encrypting controller and satellite traffic must be placed in /etc/linstor/ssl/certificates.jks and /etc/linstor/ssl/keystore.jks, respectively. The paths for these Java KeyStores are not configurable.
Securely copy the certificates.jks created for each LINSTOR satellite to /etc/linstor/certificates.jks on each satellite node. Copy the LINSTOR controller’s ssl/cerificates.jks to /etc/linstor/ssl/certificates.jks on the controller node.
Because the LINSTOR controller expects to find its keystore.jks in the /etc/linstor/ssl/keystore.jks, you can create a symlink to the existing /etc/linstor/keystore.jks on the LINSTOR controller node:
# ln -s /etc/linstor/keystore.jks /etc/linstor/ssl/keystore.jks
Configure the LINSTOR satellite to use TLS for its communication with the LINSTOR controller by creating the following linstor_satellite.toml configuration file, and restarting the LINSTOR satellite service on each satellite node:
# cat << EOF > /etc/linstor/linstor_satellite.toml
[netcom]
type="ssl"
port=3367
server_certificate="/etc/linstor/keystore.jks"
trusted_certificates="/etc/linstor/certificates.jks"
key_password="linstor"
keystore_password="linstor"
truststore_password="linstor"
ssl_protocol="TLSv1.2"
EOF
systemctl restart linstor-satellite.serviceFinally, you will need to modify the communication type configured for each LINSTOR satellite in the LINSTOR controller’s database. From any of the authenticated LINSTOR clients, run the following command to update the default interface’s communication type from “PLAIN” to “SSL” for each satellite:
# for i in linstor-sat-{0..2}; do
linstor node interface modify --communication-type ssl -p 3367 $i default
done
You should now see that all your LINSTOR satellites are connected via SSL/TLS:
# linstor node list
╭─────────────────────────────────────────────────────────────────────╮
┊ Node ┊ NodeType ┊ Addresses ┊ State ┊
╞═════════════════════════════════════════════════════════════════════╡
┊ linstor-ctrl-0 ┊ CONTROLLER ┊ 192.168.222.250:3370 (PLAIN) ┊ Online ┊
┊ linstor-sat-0 ┊ SATELLITE ┊ 192.168.222.10:3367 (SSL) ┊ Online ┊
┊ linstor-sat-1 ┊ SATELLITE ┊ 192.168.222.11:3367 (SSL) ┊ Online ┊
┊ linstor-sat-2 ┊ SATELLITE ┊ 192.168.222.12:3367 (SSL) ┊ Online ┊
╰─────────────────────────────────────────────────────────────────────╯
Concluding Thoughts
Securing LINSTOR’s REST API and the traffic between LINSTOR’s components is a good first step in hardening a system that uses LINSTOR for providing software-defined storage. Things such as LDAP authentication can further bolster a LINSTOR-based system’s security posture when an LDAP authentication server is available in an environment.
DRBD®, since version 9.2.6, is also capable of encrypting its replication traffic. Read my colleague Ryan Ronnander’s blog, “Encrypting Replication with DRBD”, to learn how to encrypt DRBD’s replicated writes in transit. Similarly, DRBD supports using LUKS encrypted volumes as backing storage to provide encryption at rest. LUKS is also supported by LINSTOR by adding a LUKS layer to the resources that LINSTOR creates.
These are some of the security best practices you can implement when using LINBIT software in secure environments. An organization’s security posture is relative to its acceptance of certain risks because absolute security is unattainable. This requires a balance between protective measures and the practicalities of operational efficiency, budget constraints, and risk tolerance. Are there security issues that you’re facing in your deployments that we haven’t considered, either in this article, or else in our user’s guides? Let us know.
