Tutorial: How to Integrate MinIO with LINSTOR to Distribute Data on Kubernetes

While LINSTOR® is on the way to becoming an industry standard as software defined block storage. MinIO stands out as the number one product in the object storage world.

LINBIT® has been working on an integration with MinIO for a long time and has infrastructures that can be used safely in business solutions.

The biggest combination of LINSTOR and MinIO is reflected in INTEL’s RSD architecture.

INTEL, which wants to offer object storage on performance with 20 servers, 4 storage in a single rack, entrusted the management of disks to LINSTOR, while using MinIO for  Object storage.

In this article, we wanted to give an example of how LINSTOR and MinIO can be combined.

If you have any questions about architecture and installation, please feel free to contact us on our slack channel.

Summary

This tutorial will show you a solution to de-couple MinIO application service and data on Kubernetes, by using LINSTOR as a distributed persistent volume instead of a local persistent volume.

1. Ubuntu Virtual Machine Setup

Create a new and updated Ubuntu x86_64 virtual machine within 2 disks, one for Ubuntu and applications, the other one will be used for MinIO data storage.

In this case:

root@minikube:~# lsb_release -r
Release:	18.04
root@minikube:~# lsblk
NAME                                                                  MAJ:MIN  RM  SIZE RO TYPE MOUNTPOINT
sda                                                                     8:0     0   30G  0 disk
vda                                                                   252:0     0   60G  0 disk
├─vda1                                                                252:1     0 59.9G  0 part /
├─vda14                                                               252:14    0    4M  0 part
└─vda15                                                               252:15    0  106M  0 part /boot/efi

2. DRBD 9.0 Setup

To install the latest DRBD® 9.0, need to add PPA from LINBIT first (ref: https://launchpad.net/~linbit/+archive/ubuntu/linbit-drbd9-stack)

root@minikube:~# add-apt-repository ppa:linbit/linbit-drbd9-stack
root@minikube:~# apt update
root@minikube:~# apt install -y linux-headers-`uname -r`
root@minikube:~# apt install -y drbd-utils drbd-dkms lvm2

Take a look at the details of DRBD 9.0:

root@minikube:~# modinfo drbd
filename:       /lib/modules/4.15.0-106-generic/updates/dkms/drbd.ko
alias:          block-major-147-*
license:        GPL
version:        9.0.23-1
description:    drbd - Distributed Replicated Block Device v9.0.23-1
author:         Philipp Reisner <[email protected]>, Lars Ellenberg <[email protected]>
srcversion:     96D47E12A7A913F21D2507E
depends:        libcrc32c
retpoline:      Y
name:           drbd
vermagic:       4.15.0-106-generic SMP mod_unload
signat:         PKCS#7
signer:
sig_key:
sig_hashalgo:   md4
parm:           enable_faults:int
parm:           fault_rate:int
parm:           fault_count:int
parm:           fault_devs:int
parm:           disable_sendpage:bool
parm:           allow_oos:DONT USE! (bool)
parm:           minor_count:Approximate number of drbd devices (1-255) (uint)
parm:           usermode_helper:string
parm:           protocol_version_min:drbd_protocol_version

3. Minikube Setup

According to the Kubernetes official documentation, install the latest minikube and kubectl both on Ubuntu.

root@minikube:~# curl -L https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 -o /usr/local/bin/minikube && chmod a+x /usr/local/bin/minikube
root@minikube:~# curl -L https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl && chmod a+x /usr/local/bin/kubectl

Considering that minikube will be running in virtual machines directly, then set minikube driver to bare-metal.

There are two pre-requirements needed to be met first (ref: https://minikube.sigs.k8s.io/docs/drivers/none/)

3.1 Install Docker

(ref: https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository)

root@minikube:~# apt install -y apt-transport-https  ca-certificates  curl gnupg-agent  software-properties-common
root@minikube:~# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
root@minikube:~# apt-key fingerprint 0EBFCD88
root@minikube:~# add-apt-repository  "deb [arch=amd64] https://download.docker.com/linux/ubuntu  $(lsb_release -cs)  stable"
root@minikube:~# apt update
root@minikube:~# apt install -y docker-ce docker-ce-cli containerd.io
root@minikube:~# docker run hello-world

3.2 Install Conntrack

root@minikube:~# apt install -y conntrack

Now, it’s time to set minikube driver and start minikube.

root@minikube:~# minikube config set driver none
root@minikube:~# minikube start

4. LINSTOR Deployment with Helm v3 Chart

Deployment of LINSTOR CSI operator is recommended. We maintain Helm charts for this, and as such suggest the use of Helm v3.

Download Helm v3 from Github (https://github.com/helm/helm/releases/latest), and copy helm to /usr/local/bin/ .

And label the nodes that will be used for LINSTOR. = hostname, in this case = minikube.

root@minikube:~# kubectl label nodes minikube linstor.linbit.com/piraeus-node=true

In this tutorial, lvm-thin will be used for back-end storage.

root@minikube:~# git clone https://github.com/piraeusdatastore/piraeus-operator
root@minikube:~# helm dependency update ./piraeus-operator/charts/piraeus
root@minikube:~# helm install piraeus-op ./piraeus-operator/charts/piraeus --set operator.nodeSet.automaticStorageType=LVMTHIN

Now docker instances are being created, and DRBD v9.0 will be injected automatically, and all unused disks will be used for LINSTOR storage-pool automatically.

root@minikube:~# lsmod | grep -i drbd
drbd_transport_tcp     24576  0
drbd                  536576  3 drbd_transport_tcp
libcrc32c              16384  6 nf_conntrack,nf_nat,dm_persistent_data,drbd,raid456,ip_vs

root@minikube:~# lsblk
NAME                                                                  MAJ:MIN  RM  SIZE RO TYPE MOUNTPOINT
sda                                                                     8:0     0   30G  0 disk
├─linstor_sda-sda_tmeta                                               253:0     0   32M  0 lvm
│ └─linstor_sda-sda-tpool                                             253:2     0   30G  0 lvm
└─linstor_sda-sda_tdata                                               253:1     0   30G  0 lvm
  └─linstor_sda-sda-tpool                                             253:2     0   30G  0 lvm

Verify this deployment via LINSTOR client, to see what happened actually.

root@minikube:~# apt install -y linstor-client

root@minikube:~# linstor node list
+--------------------------------------------------------------------------------+
| Node                       | NodeType   | Addresses                   | State  |
|================================================================================|
| minikube                   | SATELLITE  | 192.168.122.71:3366 (PLAIN) | Online |
| piraeus-op-cs-controller-0 | CONTROLLER | 172.17.0.9:3366 (PLAIN)     | Online |
+--------------------------------------------------------------------------------+

root@minikube:~# linstor storage-pool list
+--------------------------------------------------------------------------------------------------------------------+
| StoragePool          | Node     | Driver   | PoolName        | FreeCapacity | TotalCapacity | CanSnapshots | State |
|====================================================================================================================|
| DfltDisklessStorPool | minikube | DISKLESS |                 |              |               | False        | Ok    |
| sda                  | minikube | LVM_THIN | linstor_sda/sda |    27.79 GiB |     29.93 GiB | True         | Ok    |
+--------------------------------------------------------------------------------------------------------------------+

New storage-pool is created, and named as the device name automatically.

5. MinIO Setup

Create storage class for MinIO first:

root@minikube:~# cat sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
  annotations:
  storageclass.kubernetes.io/is-default-class: "true"
metadata:
  name: "linstor-csi-lvm-thin-r1"
parameters:
  autoplace: "1"
  storagePool: "sda"
provisioner: linstor.csi.linbit.com
reclaimPolicy: Delete
root@minikube:~# kubectl apply -f sc.yaml

Deploy MinIO:

root@minikube:~# cat minio.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio-deployment
spec:
  selector:
    matchLabels:
      app: minio
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: minio
    spec:
      volumes:
      - name: storage
        persistentVolumeClaim:
          claimName: minio-pv-claim
      containers:
      - name: minio
        image: minio/minio:latest
        args:
        - server
        - /data
        env:
        # Minio access key and secret key
        - name: MINIO_ACCESS_KEY
          value: "minio"
        - name: MINIO_SECRET_KEY
          value: "minio123"
        ports:
        - containerPort: 9000
          hostPort: 9000
        volumeMounts:
        - name: storage 
          mountPath: "/data"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minio-pv-claim
  labels:
    app: minio-storage-claim
spec:
  storageClassName: linstor-csi-lvm-thin-r1
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10G
---
apiVersion: v1
kind: Service
metadata:
  name: minio-service
spec:
  type: LoadBalancer
  ports:
    - port: 9000
      nodePort: 32701
      protocol: TCP
  selector:
    app: minio
  sessionAffinity: None
  type: NodePort

root@minikube:~# kubectl apply -f minio.yaml

Verify this pvc in LINSTOR.

root@minikube:~# linstor volume-definition list
+----------------------------------------------------------------------------------------------+
| ResourceName                             | VolumeNr | VolumeMinor | Size     | Gross | State |
|==============================================================================================|
| pvc-8838db2c-d65d-4b6c-8bbf-660ea7718699 | 0        | 1000        | 9.31 GiB |       | ok    |
+----------------------------------------------------------------------------------------------+

And take look at block device level in Ubuntu:

root@minikube:~# lsblk
NAME                                                                  MAJ:MIN  RM  SIZE RO TYPE MOUNTPOINT
sda                                                                     8:0     0   30G  0 disk
├─linstor_sda-sda_tmeta                                               253:0     0   32M  0 lvm
│ └─linstor_sda-sda-tpool                                             253:2     0   30G  0 lvm
│   ├─linstor_sda-sda                                                 253:3     0   30G  0 lvm
│   └─linstor_sda-pvc--8838db2c--d65d--4b6c--8bbf--660ea7718699_00000 253:4     0  9.3G  0 lvm
│     └─drbd1000                                                      147:1000  0  9.3G  0 disk /var/lib/kubelet/pods/2141f8df-d320-4bd6-8af5-418a1c5dfa0b/volumes/kubernetes.io~csi/pvc-8838db2c-d65d-4b6c-8bbf-660ea7718699/mount
└─linstor_sda-sda_tdata                                               253:1     0   30G  0 lvm
  └─linstor_sda-sda-tpool                                             253:2     0   30G  0 lvm
    ├─linstor_sda-sda                                                 253:3     0   30G  0 lvm
    └─linstor_sda-pvc--8838db2c--d65d--4b6c--8bbf--660ea7718699_00000 253:4     0  9.3G  0 lvm
      └─drbd1000                                                      147:1000  0  9.3G  0 disk /var/lib/kubelet/pods/2141f8df-d320-4bd6-8af5-418a1c5dfa0b/volumes/kubernetes.io~csi/pvc-8838db2c-d65d-4b6c-8bbf-660ea7718699/mount

In a browser, navigate to the IP address of this Ubuntu virtual machine at the exposed port 9000 or 32701, and login using the default credentials:

Access Key : minio 
Secret key : minio123

Upload some files, in this case, Fedora-Cinnamon-Live-x86_64-32-1.6.iso (1.9GiB) will be used. And check LINSTOR again.

root@minikube:~# linstor volume list
+------------------------------------------------------------------------------------------------------------------------------------+
| Node     | Resource                                 | StoragePool | VolNr | MinorNr | DeviceName    | Allocated | InUse |    State |
|====================================================================================================================================|
| minikube | pvc-8838db2c-d65d-4b6c-8bbf-660ea7718699 | sda         |     0 |    1000 | /dev/drbd1000 |  2.14 GiB | InUse | UpToDate |
+------------------------------------------------------------------------------------------------------------------------------------+

Use the official MinIO client to see files within the exact credentials above.

root@minikube:~# ./mc ls local
[2020-06-15 21:33:35 CST]      0B my-bucket/
root@minikube:~# ./mc ls local/my-bucket/
[2020-06-15 21:34:41 CST]  1.9GiB Fedora-Cinnamon-Live-x86_64-32-1.6.iso

Conclusions

Using LINBIT’s LINSTOR as a block storage orchestrator, not only replicates data to many different server nodes, but also supports disk-less mode to allow access to block storage from one node to another. Even more, when integrated with the Stork plugin it can give you features to run the pod on the same server node housing the data allowing for native storage performance.