Customer Login
Contact Us

How to use an open source SDS solution for HashiCorp Nomad

Posted on
ByJoel Zhou
linbit sds logo and nomad logo

Overview

Nomad is a simple and flexible workload orchestrator which deploys and manages containers and non-containerized applications. It does so across on-prem and clouds at scale and supports Container Storage Interface (CSI).

LINBIT SDS is an open-source software-defined storage solution and is certified as the only open-source Technology Partner Integration through CSI by HashiCorp so far.

In this quick demo, we will present one virtual machine that will run Nomad in dev mode locally.

We will show you how to set up on CentOS 8 (hostname = demo, IP = 10.43.224.23), and of course, you can choose any Linux distribution that supports Docker Engine and Nomad.

Preparation

  1. Validated LINBIT credentials (replace LINBIT_USERNAME and LINBIT_PASSWORD below)
  2. Updated CentOS 8 x86_64 virtual machine
  3. An extra virtual disk attached
  4. SELinux and firewalld are disabled.
  5. Install Docker Engine (official docs)
  6. Install Nomad (official docs)

Configuring Nomad Agent

  1. Download the latest CNI plugins, then extract to /opt/cni/bin/
  2. Support the docker driver and allow executing privileged Docker containers.
cat > /etc/nomad.d/docker-privileged.hcl <<EOL
plugin “docker” {
  config {
    allow_privileged = true
  }
}
EOL

  1. Provide a host volume, allowing a container access to the hosts /dev directory
cat > /etc/nomad.d/host-volume-dev.hcl <<EOL
client {
  host_volume “dev” {
    path = “/dev”
  }
}
EOL

  1. In order for the drbd-loader container to build DRBD or load existing modules, it needs access to the hosts /usr/src and /lib/modules respectively.
cat > /etc/nomad.d/drbd-loader-volumes.hcl <<EOL
client {
  host_volume “modules” {
    path = “/lib/modules”
    read_only = true
  }
  host_volume “kernel-src” {
    path = “/usr/src”
    read_only = true
  }
}
EOL

Running Nomad agent in dev mode locally

nomad agent -dev -config=/etc/nomad.d/ –bind 0.0.0.0 –log-level=INFO

Installing LINSTOR

  1. Nomad job for LINSTOR controller
cat > /tmp/controller.nomad <<EOL
job “linstor-controller” {
  datacenters = [“dc1”]
  type = “service”

  group “linstor-controller” {
    network {
      mode = “bridge”
      port “linstor-api” {
        static = 3370
        to = 3370
      }
    }

    task “linstor-controller” {
      driver = “docker”
      config {
        image = “drbd.io/linstor-controller:v1.14.0”

        auth {
          username = “LINBIT_USERNAME”
          password = “LINBIT_PASSWORD”
          server_address = “drbd.io”
        }

        mount {
          type = “bind”
          source = “local”
          target = “/etc/linstor”
        }
      }

      resources {
        cpu    = 1000
        memory = 1000
      }
    }
  }
}
EOL

[root@demo ~]# nomad job run /tmp/controller.nomad

  1. Nomad job for LINSTOR satellite
cat > /tmp/satellite.nomad <<EOL
job “linstor-satellite” {
  datacenters = [“dc1”]
  type = “system”

  group “satellite” {
    network {
      mode = “host”
    }

    volume “dev” {
      type = “host”
      source = “dev”
    }

    task “linstor-satellite” {
      driver = “docker”

      config {
        image = “drbd.io/linstor-satellite:v1.14.0”

        auth {
          username = “LINBIT_USERNAME”
          password = “LINBIT_PASSWORD”
          server_address = “drbd.io”
        }

        privileged = true
        network_mode = “host”
      }

      volume_mount {
        volume = “dev”
        destination = “/dev”
      }

      resources {
        cpu    = 1000
        memory = 1000
      }
    }

    task “drbd-loader” {
      driver = “docker”
      lifecycle {
        hook = “prestart”
      }

      config {
        image = “drbd.io/drbd9-rhel8:v9.1.3”

        privileged = true
        auth {
          username = “LINBIT_USERNAME”
          password = “LINBIT_PASSWORD”
          server_address = “drbd.io”
        }
      }

      env {
        LB_HOW = “shipped_modules”
      }

      volume_mount {
        volume = “kernel-src”
        destination = “/usr/src”
      }
      volume_mount {
        volume = “modules”
        destination = “/lib/modules”
      }
    }

    volume “modules” {
      type = “host”
      source = “modules”
      read_only = true
    }

    volume “kernel-src” {
      type = “host”
      source = “kernel-src”
      read_only = true
    }
  }
}
EOL

[root@demo ~]# nomad job run /tmp/satellite.nomad

Setting up LINSTOR

You can run LINSTOR CLI in three different ways – choose any one of them that you like:

  1. Install linstor-client on CentOS 8 via DNF
  2. Install linstor-client as dedicated container
  3. Run linstor-client in linstor-controller container, no installation needed

Adding node to the LINSTOR cluster

[root@demo ~]# linstor node create demo 10.43.224.23

Creating LINSTOR storage-pool

[root@demo ~]# linstor physical-storage create-device-pool –pool-name MyThin –storage-pool MySP lvmthin demo /dev/vdb

Setting up volume for application via CSI

  1. Nomad job for LINSTOR CSI driver
cat > /tmp/csi.nomad <<EOL
job “linstor-csi” {
  datacenters = [“dc1”]
  type = “system”

  group “csi” {
    network {
      mode = “bridge”
    }

    task “csi-plugin” {
      driver = “docker”
      config {
        image = “drbd.io/linstor-csi:v0.13.1”

        auth {
          username = “LINBIT_USERNAME”
          password = “LINBIT_PASSWORD”
          server_address = “drbd.io”
        }

        args = [
          “–csi-endpoint=unix://csi/csi.sock”,
          “–node=demo”,
          “–linstor-endpoint=http://10.43.224.23:3370”,
          “–log-level=info”
        ]

        privileged = true
      }

      csi_plugin {
        id = “linstor.csi.linbit.com”
        type = “monolith”
        mount_dir = “/csi”
      }

      resources {
        cpu    = 1000
        memory = 1000
      }
    }
  }
}
EOL

[root@demo ~]# nomad job run /tmp/csi.nomad

  1. Creating 2GiB volume
cat > /tmp/volume.nomad <<EOL
id = “vol1”
name = “vol1”
type = “csi”
plugin_id = “linstor.csi.linbit.com”
capacity_min = “2GiB”
capacity_max = “2GiB”
capability {
  access_mode = “single-node-writer”
  attachment_mode = “file-system”
}
mount_options {
  fs_type = “ext4”
}
parameters {
  “resourceGroup” = “DfltRscGrp”,
  “storagePool” = “MySP”,
  “autoPlace” = “1”
}
EOL

[root@demo ~]# nomad volume create /tmp/volume.nomad

  1. Creating Nginx demo app
cat > /tmp/nginx.nomad <<EOL
job “nginx” {
  datacenters = [“dc1”]
  type = “service”

  group “web” {
    count = 1
    volume “vol1” {
      type = “csi”
      source = “vol1”
      attachment_mode = “file-system”
      access_mode = “single-node-writer”
    }
    network {
      port “http” {
        to = 80
      }
    }

    task “nginx” {
      driver = “docker”

      config {
        image = “nginx:stable”
        ports = [“http”]
      }
      volume_mount {
        volume      = “vol1”
        destination = “/usr/share/nginx/html/”
        read_only   = false
      }
      resources {
        cpu    = 1000
        memory = 1000
      }
    }
  }
}
EOL

[root@demo ~]# nomad job run /tmp/nginx.nomad

Congratulations! Now 2GiB volume should be attached to the Nginx container as web root directory /usr/share/nginx/html/, backed by /dev/drbd1000 locally, no replicas in this setup.

[root@demo ~]# lsblk

For detailed information, please take a look at LINSTOR docs.

Contact us in case you need any support.

Share on facebook
Share on twitter
Share on linkedin
Share on reddit
Share on whatsapp
Share on vk
Share on email

Share this post

Recent posts
Joel Zhou

Joel Zhou

Talk to us

LINBIT is committed to protecting and respecting your privacy, and we’ll only use your personal information to administer your account and to provide the products and services you requested from us. From time to time, we would like to contact you about our products and services, as well as other content that may be of interest to you. If you consent to us contacting you for this purpose, please tick above to say how you would like us to contact you.

You can unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.

By clicking submit below, you consent to allow LINBIT to store and process the personal information submitted above to provide you the content requested.

Talk to us

LINBIT is committed to protecting and respecting your privacy, and we’ll only use your personal information to administer your account and to provide the products and services you requested from us. From time to time, we would like to contact you about our products and services, as well as other content that may be of interest to you. If you consent to us contacting you for this purpose, please tick above to say how you would like us to contact you.

You can unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.

By clicking submit below, you consent to allow LINBIT to store and process the personal information submitted above to provide you the content requested.

As the world’s leading provider of Software-Defined Storage, High Availability, and Disaster Recovery software, LINBIT adds server clustering capabilities to any containerized, virtualized, or bare metal environment.
The words LINSTOR®, DRBD®, LINBIT®, and the logo LINSTOR®, DRBD®, and LINBIT® are trademarks or registered trademarks of LINBIT in Austria, the United States and other countries.
Copyright © 2022 LINBIT ALL rights reserved.
Legal
Open Source
Terms & Conditions
Legal Notice
Privacy
Resources
Connect
Login
Request a demo
Twitter Linkedin Facebook Youtube Github Wikipedia-w Slack