Overview

Nomad is a simple and flexible workload orchestrator which deploys and manages containers and non-containerized applications. It does so across on-prem and clouds at scale and supports Container Storage Interface (CSI).

LINBIT SDS is an open-source software-defined storage solution and is certified as the only open-source Technology Partner Integration through CSI by HashiCorp so far.

In this quick demo, we will present one virtual machine that will run Nomad in dev mode locally.

We will show you how to set up on CentOS 8 (hostname = demo, IP = 10.43.224.23), and of course, you can choose any Linux distribution that supports Docker Engine and Nomad.

Preparation

1. Validated LINBIT credentials (replace LINBIT_USERNAME and LINBIT_PASSWORD below)
2. Updated CentOS 8 x86_64 virtual machine
3. An extra virtual disk attached
4. SELinux and firewalls are disabled.
5. Install Docker Engine (official docs)
6. Install Nomad (official docs)

Configuring Nomad Agent

1. Download the latest CNI plugins, then extract to /opt/cni/bin/
2. Support the docker driver and allow executing privileged Docker containers.

cat > /etc/nomad.d/docker-privileged.hcl <<EOL
plugin “docker” {
  config {
    allow_privileged = true
  }
}
EOL

3. Provide a host volume, allowing a container access to the hosts /dev directory

cat > /etc/nomad.d/host-volume-dev.hcl <<EOL
client {
  host_volume “dev” {
    path = “/dev”
  }
}
EOL

4. In order for the drbd-loader container to build DRBD or load existing modules, it needs access to the hosts /usr/src and /lib/modules respectively.

cat > /etc/nomad.d/drbd-loader-volumes.hcl <<EOL
client {
  host_volume “modules” {
    path = “/lib/modules”
    read_only = true
  }
  host_volume “kernel-src” {
    path = “/usr/src”
    read_only = true
  }
}
EOL

Running Nomad agent in dev mode locally

nomad agent -dev -config=/etc/nomad.d/ –bind 0.0.0.0 –log-level=INFO

Installing LINSTOR

1. Nomad job for LINSTOR controller

cat > /tmp/controller.nomad <<EOL
job “linstor-controller” {
  datacenters = [“dc1”]
  type = “service”

  group “linstor-controller” {
    network {
      mode = “bridge”
      port “linstor-api” {
        static = 3370
        to = 3370
      }
    }

    task “linstor-controller” {
      driver = “docker”
      config {
        image = “drbd.io/linstor-controller:v1.14.0”

        auth {
          username = “LINBIT_USERNAME”
          password = “LINBIT_PASSWORD”
          server_address = “drbd.io”
        }

        mount {
          type = “bind”
          source = “local”
          target = “/etc/linstor”
        }
      }

      resources {
        cpu    = 1000
        memory = 1000
      }
    }
  }
}
EOL

[root@demo ~]# nomad job run /tmp/controller.nomad

2. Nomad job for LINSTOR satellite

cat > /tmp/satellite.nomad <<EOL
job “linstor-satellite” {
  datacenters = [“dc1”]
  type = “system”

  group “satellite” {
    network {
      mode = “host”
    }

    volume “dev” {
      type = “host”
      source = “dev”
    }

    task “linstor-satellite” {
      driver = “docker”

      config {
        image = “drbd.io/linstor-satellite:v1.14.0”

        auth {
          username = “LINBIT_USERNAME”
          password = “LINBIT_PASSWORD”
          server_address = “drbd.io”
        }

        privileged = true
        network_mode = “host”
      }

      volume_mount {
        volume = “dev”
        destination = “/dev”
      }

      resources {
        cpu    = 1000
        memory = 1000
      }
    }

    task “drbd-loader” {
      driver = “docker”
      lifecycle {
        hook = “prestart”
      }

      config {
        image = “drbd.io/drbd9-rhel8:v9.1.3”

        privileged = true
        auth {
          username = “LINBIT_USERNAME”
          password = “LINBIT_PASSWORD”
          server_address = “drbd.io”
        }
      }

      env {
        LB_HOW = “shipped_modules”
      }

      volume_mount {
        volume = “kernel-src”
        destination = “/usr/src”
      }
      volume_mount {
        volume = “modules”
        destination = “/lib/modules”
      }
    }

    volume “modules” {
      type = “host”
      source = “modules”
      read_only = true
    }

    volume “kernel-src” {
      type = “host”
      source = “kernel-src”
      read_only = true
    }
  }
}
EOL

[root@demo ~]# nomad job run /tmp/satellite.nomad

Setting up LINSTOR

You can run LINSTOR CLI in three different ways – choose any one of them that you like:

1. Install linstor-client on CentOS 8 via DNF
2. Install linstor-client as dedicated container
3. Run linstor-client in linstor-controller container, no installation is needed

Adding node to the LINSTOR cluster

[root@demo ~]# linstor node create demo 10.43.224.23

Creating LINSTOR storage-pool

[root@demo ~]# linstor physical-storage create-device-pool –pool-name MyThin –storage-pool MySP lvmthin demo /dev/vdb

Setting up volume for application via CSI

1. Nomad job for LINSTOR CSI driver

cat > /tmp/csi.nomad <<EOL
job “linstor-csi” {
  datacenters = [“dc1”]
  type = “system”

  group “csi” {
    network {
      mode = “bridge”
    }

    task “csi-plugin” {
      driver = “docker”
      config {
        image = “drbd.io/linstor-csi:v0.13.1”

        auth {
          username = “LINBIT_USERNAME”
          password = “LINBIT_PASSWORD”
          server_address = “drbd.io”
        }

        args = [
          “–csi-endpoint=unix://csi/csi.sock”,
          “–node=demo”,
          “–linstor-endpoint=http://10.43.224.23:3370”,
          “–log-level=info”
        ]

        privileged = true
      }

      csi_plugin {
        id = “linstor.csi.linbit.com”
        type = “monolith”
        mount_dir = “/csi”
      }

      resources {
        cpu    = 1000
        memory = 1000
      }
    }
  }
}
EOL

[root@demo ~]# nomad job run /tmp/csi.nomad

2. Creating 2GiB volume

cat > /tmp/volume.nomad <<EOL
id = “vol1”
name = “vol1”
type = “csi”
plugin_id = “linstor.csi.linbit.com”
capacity_min = “2GiB”
capacity_max = “2GiB”
capability {
  access_mode = “single-node-writer”
  attachment_mode = “file-system”
}
mount_options {
  fs_type = “ext4”
}
parameters {
  “resourceGroup” = “DfltRscGrp”,
  “storagePool” = “MySP”,
  “autoPlace” = “1”
}
EOL

[root@demo ~]# nomad volume create /tmp/volume.nomad

3. Creating Nginx demo app

cat > /tmp/nginx.nomad <<EOL
job “nginx” {
  datacenters = [“dc1”]
  type = “service”

  group “web” {
    count = 1
    volume “vol1” {
      type = “csi”
      source = “vol1”
      attachment_mode = “file-system”
      access_mode = “single-node-writer”
    }
    network {
      port “http” {
        to = 80
      }
    }

    task “nginx” {
      driver = “docker”

      config {
        image = “nginx:stable”
        ports = [“http”]
      }
      volume_mount {
        volume      = “vol1”
        destination = “/usr/share/nginx/html/”
        read_only   = false
      }
      resources {
        cpu    = 1000
        memory = 1000
      }
    }
  }
}
EOL

[root@demo ~]# nomad job run /tmp/nginx.nomad

Congratulations! Now 2GiB volume should be attached to the Nginx container as web root directory /usr/share/nginx/html/, backed by /dev/drbd1000 locally, no replicas in this setup.

[root@demo ~]# lsblk

For detailed information, please take a look at LINSTOR docs.

Contact us in case you need any support.