Nomad as Sysbox Containers Orchestrator

3 min readJan 16, 2022

How to use System Containers on HashiCorp Nomad

This short article introduces the use of Sysbox (by Nestybox) to deploy system containers on HashiCorp Nomad.

With Sysbox containers, Nomad orchestrates quickly and efficiently VM-like workloads, using unprivileged containers without using QEMU.

To know more about Sysbox Containers, read the article What’s Sysbox by Nestybox?

In following procedures, the Linux host is installed with Ubuntu 20.04.

Sysbox Installation

After downloading the software package from the official release page, the Sysbox package is installed with the apt install command:

$ sudo apt install ./sysbox-ce_0.4.1-0.ubuntu-focal_amd64.deb

See the complete installation guide for Docker host on the following page: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-package.md#available-sysbox-packages

Nomad Installation

The Nomad package is installed with the apt install command after configuring the official HashiCorp Linux repository:

$ curl -fsSL https://apt.releases.hashicorp.com/gpg \
    | sudo apt-key add -
$ sudo apt-add-repository "deb [arch=amd64] \
    https://apt.releases.hashicorp.com \
    $(lsb_release -cs) main"
$ sudo apt update
$ sudo apt install nomad

See a detailed tutorial to install Nomad on the following page:
https://learn.hashicorp.com/tutorials/nomad/get-started-install

Nomad Configuration

The deployed configuration file is located in /etc/nomad.d/nomad.hcl.
By default, it enables server and client agent modes.

First of all, the Docker driver needs to allow sysbox-runcruntime. To do so, add a configuration file in the folder /etc/nomad.d.

docker.hcl

plugin "docker" {
  config {
    allow_runtimes = ["runc", "sysbox-runc"]
  }
}

See the official documentation for more details:
https://www.nomadproject.io/docs/drivers/docker#allow_runtimes

Then, restart the Nomad service.

$ sudo systemctl restart nomad

So, Nomad is ready to orchestrate the Sysbox containers.

Nomad Job

Let’s define a job specification.

First, create a job specification file.

$ nomad job init -short sysbox.nomad

Next, edit the created file and add the optionruntime = "sysbox-runc" in the Docker config stanza. The selected container image contains systemd, Docker, and SSH services.

See the related Dockerfile in the following GitHub project:
https://github.com/nestybox/dockerfiles/blob/master/ubuntu-bionic-systemd-docker/Dockerfile

sysbox.nomad

job "example" {
  datacenters = ["dc1"]  group "sysbox" {
    network {
      port "ssh" {
        to = 22
      }
    }  task "system" {
      driver = "docker"      config {
        image   = "registry.nestybox.com/nestybox/ubuntu-bionic-systemd-docker"
        runtime = "sysbox-runc"
        ports   = ["ssh"]
      }      resources {
        cpu    = 1000
        memory = 1024
      }
    }
  }
}

Thus, let’s run the job.

$ nomad job run sysbox.nomad

The container image is downloaded, and the Docker driver delegates to Sysbox the spawning of the container.

List the running container with docker ps command to retrieve the exposed port for SSH. Then, connect to it (both login and password are admin).

$ ssh admin@172.19.36.50 -p 30971
admin@172.19.36.50’s password:Welcome to Ubuntu 18.04.4 LTS (GNU/Linux 5.4.0–91-generic x86_64) * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
...
admin@e8442410576d:~$

Launch hostnamectl to get the details of the running “system”.

admin@e8442410576d:~$ hostnamectl
  Static hostname: e8442410576d
        Icon name: computer-container
          Chassis: container
       Machine ID: 32c2074b9f9d4ed684d9265d98be5e45
          Boot ID: cff163f3343e4026ae3c78e06f8974a9
   Virtualization: container-other
 Operating System: Ubuntu 18.04.4 LTS
           Kernel: Linux 5.4.0–91-generic
     Architecture: x86–64

As Docker is installed in the selected Sysbox container image, let’s launchhello-world image.

admin@e8442410576d:~$ docker run hello-world
Unable to find image ‘hello-world:latest’ locally
...
Status: Downloaded newer image for hello-world:latestHello from Docker!
This message shows that your installation appears to be working correctly.To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the “hello-world” image from the Docker Hub.
 (amd64)
 3. The Docker daemon created a new container from that image which runs the
 executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
 to your terminal.To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bashShare images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/For more examples and ideas, visit:
https://docs.docker.com/get-started/

Finally, to exit the Sysbox container, execute logout command. Then, stop example job and stop nomad service to clean-up.

$ nomad job stop -purge example
$ sudo systemctl stop nomad.service

References

Nestybox home page
Sysbox repository
HashiCorp Nomad