How to use an open source SDS solution for HashiCorp Nomad

Overview

Nomad is a simple and flexible workload orchestrator which deploys and manages containers and non-containerized applications. It does so across on-prem and clouds at scale and supports Container Storage Interface (CSI).

LINBIT SDS is an open-source software-defined storage solution and is certified as the only open-source Technology Partner Integration through CSI by HashiCorp so far.

In this quick demo, we will present one virtual machine that will run Nomad in dev mode locally.

We will show you how to set up on CentOS 8 (hostname = demo, IP = 10.43.224.23), and of course, you can choose any Linux distribution that supports Docker Engine and Nomad.

Preparation

Validated LINBIT credentials (replace LINBIT_USERNAME and LINBIT_PASSWORD below)
Updated CentOS 8 x86_64 virtual machine
An extra virtual disk attached
SELinux and firewalld are disabled.
Install Docker Engine (official docs)
Install Nomad (official docs)

Configuring Nomad Agent

Download the latest CNI plugins, then extract to /opt/cni/bin/
Support the docker driver and allow executing privileged Docker containers.

cat > /etc/nomad.d/docker-privileged.hcl <<EOL
plugin “docker” {
config {
allow_privileged = true
}
}
EOL

Provide a host volume, allowing a container access to the hosts /dev directory

cat > /etc/nomad.d/host-volume-dev.hcl <<EOL
client {
host_volume “dev” {
path = “/dev”
}
}
EOL

In order for the drbd-loader container to build DRBD or load existing modules, it needs access to the hosts /usr/src and /lib/modules respectively.

cat > /etc/nomad.d/drbd-loader-volumes.hcl <<EOL
client {
host_volume “modules” {
path = “/lib/modules”
read_only = true
}
host_volume “kernel-src” {
path = “/usr/src”
read_only = true
}
}
EOL

Running Nomad agent in dev mode locally

nomad agent -dev -config=/etc/nomad.d/ –bind 0.0.0.0 –log-level=INFO

Installing LINSTOR

Nomad job for LINSTOR controller

cat > /tmp/controller.nomad <<EOL
job “linstor-controller” {
datacenters = [“dc1”]
type = “service”

group “linstor-controller” {
network {
mode = “bridge”
port “linstor-api” {
static = 3370
to = 3370
}
}

task “linstor-controller” {
driver = “docker”
config {
image = “drbd.io/linstor-controller:v1.14.0”

auth {
username = “LINBIT_USERNAME”
password = “LINBIT_PASSWORD”
server_address = “drbd.io”
}

mount {
type = “bind”
source = “local”
target = “/etc/linstor”
}
}

resources {
cpu = 1000
memory = 1000
}
}
}
}
EOL

[root@demo ~]# nomad job run /tmp/controller.nomad

Nomad job for LINSTOR satellite

cat > /tmp/satellite.nomad <<EOL
job “linstor-satellite” {
datacenters = [“dc1”]
type = “system”

group “satellite” {
network {
mode = “host”
}

volume “dev” {
type = “host”
source = “dev”
}

task “linstor-satellite” {
driver = “docker”

config {
image = “drbd.io/linstor-satellite:v1.14.0”

auth {
username = “LINBIT_USERNAME”
password = “LINBIT_PASSWORD”
server_address = “drbd.io”
}

privileged = true
network_mode = “host”
}

volume_mount {
volume = “dev”
destination = “/dev”
}

resources {
cpu = 1000
memory = 1000
}
}

task “drbd-loader” {
driver = “docker”
lifecycle {
hook = “prestart”
}

config {
image = “drbd.io/drbd9-rhel8:v9.1.3”

privileged = true
auth {
username = “LINBIT_USERNAME”
password = “LINBIT_PASSWORD”
server_address = “drbd.io”
}
}

env {
LB_HOW = “shipped_modules”
}

volume_mount {
volume = “kernel-src”
destination = “/usr/src”
}
volume_mount {
volume = “modules”
destination = “/lib/modules”
}
}

volume “modules” {
type = “host”
source = “modules”
read_only = true
}

volume “kernel-src” {
type = “host”
source = “kernel-src”
read_only = true
}
}
}
EOL

[root@demo ~]# nomad job run /tmp/satellite.nomad

Setting up LINSTOR

You can run LINSTOR CLI in three different ways – choose any one of them that you like:

Install linstor-client on CentOS 8 via DNF
Install linstor-client as dedicated container
Run linstor-client in linstor-controller container, no installation needed

Adding node to the LINSTOR cluster

[root@demo ~]# linstor node create demo 10.43.224.23

Creating LINSTOR storage-pool

[root@demo ~]# linstor physical-storage create-device-pool –pool-name MyThin –storage-pool MySP lvmthin demo /dev/vdb

Setting up volume for application via CSI

Nomad job for LINSTOR CSI driver

cat > /tmp/csi.nomad <<EOL
job “linstor-csi” {
datacenters = [“dc1”]
type = “system”

group “csi” {
network {
mode = “bridge”
}

task “csi-plugin” {
driver = “docker”
config {
image = “drbd.io/linstor-csi:v0.13.1”

auth {
username = “LINBIT_USERNAME”
password = “LINBIT_PASSWORD”
server_address = “drbd.io”
}

args = [
“–csi-endpoint=unix://csi/csi.sock”,
“–node=demo”,
“–linstor-endpoint=http://10.43.224.23:3370”,
“–log-level=info”
]

privileged = true
}

csi_plugin {
id = “linstor.csi.linbit.com”
type = “monolith”
mount_dir = “/csi”
}

resources {
cpu = 1000
memory = 1000
}
}
}
}
EOL

[root@demo ~]# nomad job run /tmp/csi.nomad

Creating 2GiB volume

cat > /tmp/volume.nomad <<EOL
id = “vol1”
name = “vol1”
type = “csi”
plugin_id = “linstor.csi.linbit.com”
capacity_min = “2GiB”
capacity_max = “2GiB”
capability {
access_mode = “single-node-writer”
attachment_mode = “file-system”
}
mount_options {
fs_type = “ext4”
}
parameters {
“resourceGroup” = “DfltRscGrp”,
“storagePool” = “MySP”,
“autoPlace” = “1”
}
EOL

[root@demo ~]# nomad volume create /tmp/volume.nomad

Creating Nginx demo app

cat > /tmp/nginx.nomad <<EOL
job “nginx” {
datacenters = [“dc1”]
type = “service”

group “web” {
count = 1
volume “vol1” {
type = “csi”
source = “vol1”
attachment_mode = “file-system”
access_mode = “single-node-writer”
}
network {
port “http” {
to = 80
}
}

task “nginx” {
driver = “docker”

config {
image = “nginx:stable”
ports = [“http”]
}
volume_mount {
volume = “vol1”
destination = “/usr/share/nginx/html/”
read_only = false
}
resources {
cpu = 1000
memory = 1000
}
}
}
}
EOL

[root@demo ~]# nomad job run /tmp/nginx.nomad

Congratulations! Now 2GiB volume should be attached to the Nginx container as web root directory /usr/share/nginx/html/, backed by /dev/drbd1000 locally, no replicas in this setup.

[root@demo ~]# lsblk

For detailed information, please take a look at LINSTOR docs.

How to use an open source SDS solution for HashiCorp Nomad

Overview

Preparation

Configuring Nomad Agent

Running Nomad agent in dev mode locally

Installing LINSTOR

Setting up LINSTOR

Setting up volume for application via CSI

Like? Share it with the world.

The words LINSTOR®, DRBD®, LINBIT®, and the logo LINSTOR®, DRBD®, and LINBIT® are trademarks or registered trademarks of LINBIT in Austria, the United States and other countries.

Copyright © 2021 LINBIT All rights reserved.

Legal

Resources

Connect