Overview
Nomad is a simple and flexible workload orchestrator which deploys and manages containers and non-containerized applications. It does so across on-prem and clouds at scale and supports Container Storage Interface (CSI).
LINBIT SDS is an open-source software-defined storage solution and is certified as the only open-source Technology Partner Integration through CSI by HashiCorp so far.
In this quick demo, we will present one virtual machine that will run Nomad in dev mode locally.
We will show you how to set up on CentOS 8 (hostname = demo, IP = 10.43.224.23), and of course, you can choose any Linux distribution that supports Docker Engine and Nomad.
Preparation
- Validated LINBIT credentials (replace LINBIT_USERNAME and LINBIT_PASSWORD below)
- Updated CentOS 8 x86_64 virtual machine
- An extra virtual disk attached
- SELinux and firewalld are disabled.
- Install Docker Engine (official docs)
- Install Nomad (official docs)
Configuring Nomad Agent
- Download the latest CNI plugins, then extract to /opt/cni/bin/
- Support the docker driver and allow executing privileged Docker containers.
| cat > /etc/nomad.d/docker-privileged.hcl <<EOL plugin “docker” { config { allow_privileged = true } } EOL |
- Provide a host volume, allowing a container access to the hosts /dev directory
| cat > /etc/nomad.d/host-volume-dev.hcl <<EOL client { host_volume “dev” { path = “/dev” } } EOL |
- In order for the drbd-loader container to build DRBD or load existing modules, it needs access to the hosts /usr/src and /lib/modules respectively.
| cat > /etc/nomad.d/drbd-loader-volumes.hcl <<EOL client { host_volume “modules” { path = “/lib/modules” read_only = true } host_volume “kernel-src” { path = “/usr/src” read_only = true } } EOL |
Running Nomad agent in dev mode locally
| nomad agent -dev -config=/etc/nomad.d/ –bind 0.0.0.0 –log-level=INFO |
Installing LINSTOR
- Nomad job for LINSTOR controller
| cat > /tmp/controller.nomad <<EOL job “linstor-controller” { datacenters = [“dc1”] type = “service” group “linstor-controller” { network { mode = “bridge” port “linstor-api” { static = 3370 to = 3370 } } task “linstor-controller” { driver = “docker” config { image = “drbd.io/linstor-controller:v1.14.0” auth { username = “LINBIT_USERNAME” password = “LINBIT_PASSWORD” server_address = “drbd.io” } mount { type = “bind” source = “local” target = “/etc/linstor” } } resources { cpu = 1000 memory = 1000 } } } } EOL |
| [root@demo ~]# nomad job run /tmp/controller.nomad |
- Nomad job for LINSTOR satellite
| cat > /tmp/satellite.nomad <<EOL job “linstor-satellite” { datacenters = [“dc1”] type = “system” group “satellite” { network { mode = “host” } volume “dev” { type = “host” source = “dev” } task “linstor-satellite” { driver = “docker” config { image = “drbd.io/linstor-satellite:v1.14.0” auth { username = “LINBIT_USERNAME” password = “LINBIT_PASSWORD” server_address = “drbd.io” } privileged = true network_mode = “host” } volume_mount { volume = “dev” destination = “/dev” } resources { cpu = 1000 memory = 1000 } } task “drbd-loader” { driver = “docker” lifecycle { hook = “prestart” } config { image = “drbd.io/drbd9-rhel8:v9.1.3” privileged = true auth { username = “LINBIT_USERNAME” password = “LINBIT_PASSWORD” server_address = “drbd.io” } } env { LB_HOW = “shipped_modules” } volume_mount { volume = “kernel-src” destination = “/usr/src” } volume_mount { volume = “modules” destination = “/lib/modules” } } volume “modules” { type = “host” source = “modules” read_only = true } volume “kernel-src” { type = “host” source = “kernel-src” read_only = true } } } EOL |
| [root@demo ~]# nomad job run /tmp/satellite.nomad |
Setting up LINSTOR
You can run LINSTOR CLI in three different ways – choose any one of them that you like:
- Install linstor-client on CentOS 8 via DNF
- Install linstor-client as dedicated container
- Run linstor-client in linstor-controller container, no installation needed
Adding node to the LINSTOR cluster
| [root@demo ~]# linstor node create demo 10.43.224.23 |
Creating LINSTOR storage-pool
| [root@demo ~]# linstor physical-storage create-device-pool –pool-name MyThin –storage-pool MySP lvmthin demo /dev/vdb |
Setting up volume for application via CSI
- Nomad job for LINSTOR CSI driver
| cat > /tmp/csi.nomad <<EOL job “linstor-csi” { datacenters = [“dc1”] type = “system” group “csi” { network { mode = “bridge” } task “csi-plugin” { driver = “docker” config { image = “drbd.io/linstor-csi:v0.13.1” auth { username = “LINBIT_USERNAME” password = “LINBIT_PASSWORD” server_address = “drbd.io” } args = [ “–csi-endpoint=unix://csi/csi.sock”, “–node=demo”, “–linstor-endpoint=http://10.43.224.23:3370”, “–log-level=info” ] privileged = true } csi_plugin { id = “linstor.csi.linbit.com” type = “monolith” mount_dir = “/csi” } resources { cpu = 1000 memory = 1000 } } } } EOL |
| [root@demo ~]# nomad job run /tmp/csi.nomad |
- Creating 2GiB volume
| cat > /tmp/volume.nomad <<EOL id = “vol1” name = “vol1” type = “csi” plugin_id = “linstor.csi.linbit.com” capacity_min = “2GiB” capacity_max = “2GiB” capability { access_mode = “single-node-writer” attachment_mode = “file-system” } mount_options { fs_type = “ext4” } parameters { “resourceGroup” = “DfltRscGrp”, “storagePool” = “MySP”, “autoPlace” = “1” } EOL |
| [root@demo ~]# nomad volume create /tmp/volume.nomad |
- Creating Nginx demo app
| cat > /tmp/nginx.nomad <<EOL job “nginx” { datacenters = [“dc1”] type = “service” group “web” { count = 1 volume “vol1” { type = “csi” source = “vol1” attachment_mode = “file-system” access_mode = “single-node-writer” } network { port “http” { to = 80 } } task “nginx” { driver = “docker” config { image = “nginx:stable” ports = [“http”] } volume_mount { volume = “vol1” destination = “/usr/share/nginx/html/” read_only = false } resources { cpu = 1000 memory = 1000 } } } } EOL |
| [root@demo ~]# nomad job run /tmp/nginx.nomad |
Congratulations! Now 2GiB volume should be attached to the Nginx container as web root directory /usr/share/nginx/html/, backed by /dev/drbd1000 locally, no replicas in this setup.
| [root@demo ~]# lsblk |
For detailed information, please take a look at LINSTOR docs.
Contact us in case you need any support.
