What’s Sysbox by Nestybox?
Streamline your IT by running all your workloads in containers
Before explaining what’s Sysbox, let’s have a short, simplified, and illustrated story of the containers orchestrated by Kubernetes on Linux.
Where is located Nestybox Sysbox in the container’s ecosystem?
Docker
In the beginning, Kubernetes delegated container management to Docker.
Open Container Initiative
Then, some companies that were actors in the Linux container ecosystem (including Docker and CoreOS) created the Open Container Initiative as a project of the Linux Foundation. This project aims to define open standards around containers under Linux: mainly, OCI Runtime Specifications for the definition of a runtime container and OCI Image Specifications for the format of an image.
Thus, Docker delegates, via the OCI Runtime Specs, the spawning, and running of the containers to the container runtime runc (as a reference implementation of the OCI specs).
Container Runtime Interface
Then, to simplify the implementation of other container managers for Kubernetes (such as rkt, which is no longer developed today), the Kubernetes project decided to create an interface called Container Runtime Interface for the delegation of container management.
Therefore, as this diagram shows, rkt developers implement the rklet module to support the CRI, and Kubernetes developers implement the dockershim module.
CRI-O and containerd
Today, the ecosystem of container managers is as follows:
- CRI-O which is an implementation working specifically for Kubernetes
- containerd which works with Docker and with its CRI plugin to work with Kubernetes
The Kubernetes community decided to deprecate the dockershim module: https://kubernetes.io/blog/2020/12/08/kubernetes-1-20-release-announcement/#dockershim-deprecation
Not long ago, the Kubernetes community opens a survey:
Dockershim removal is coming. Are you ready? https://kubernetes.io/blog/2021/11/12/are-you-ready-for-dockershim-removal/
Command-line tools
To use all this bestiary, different command-line tools are available such as:
- kubectl to communicate with the Kube API.
- crictl to communicate with the CRI interface.
- Docker or nerdctl (a Docker compatible implementation) and Podman to manage containers.
Container Runtime alternatives
Thanks to the OCI specs, alternatives to runc have emerged, some examples of which are shown in the following diagram: gVisor and KataContainers.
The goal of these two alternatives is to secure by isolating the container workload from the host system:
- either with an applicative layer for gVisor
- either with a microVM for KataContainer
Sysbox — Container Runtime
Finally, let’s leave a place between runc and gVisor to accommodate sysbox.
So, Sysbox is a next-generation open-source container runtime developed and supported by Nestybox that works below Docker, Containerd, and Kubernetes.
Good news, no need to learn new tools!
Thus, the tools that you’re using with Docker and Kubernetes can be used to manage Sysbox containers.
Sysbox — System Container
What are the key concepts of Sysbox?
First, Sysbox allows the container workload isolation using Linux user namespaces. Thus, the containers generated by Sysbox are natively unprivileged containers. Additionally, the initial mounts from the host are locked while the container is running.
Then, Sysbox allows the container to act like a VM. Be careful, Sysbox doesn’t run a VM (as KubeVirt could do), but only the workloads that run on a physical machine or a virtual machine. What is important is that this is possible without using a complex image, without a sophisticated entry point or special volume mounts.
To summarize, Sysbox empowers containers to run seamlessly and securely software such as systemd, Docker, Kubernetes, and any legacy application that runs on Linux.
Sysbox — Use Cases
Sysbox can be used to secure and simplify CI/CD pipelines that run on Docker or Kubernetes, such as following tasks:
- Build Container Images inside an unprivileged Sysbox container.
- Tests Configurations and Deployments using Kubernetes, Nomad, Docker Compose, or Docker Swarm inside a Sysbox container.
Sysbox can be used as well to deploy fast Kubernetes Clusters and containerized Dev Environments.
Dedicated articles explain these two last usages:
- Kubernetes-in-Docker with kubeadm and Sysbox
- Kubernetes-in-Kubernetes with kubeadm and Sysbox
- Shift your Workspaces to the Cloud — “It works on my browser!”
At last, Sysbox exists in two editions (see Features Comparison page):
Sysbox CE (Community Edition) and Sysbox EE (Enterprise Edition).
In following procedures, the Linux host is installed with Ubuntu 20.04.
Sysbox Image
To build an image that can be used by Sysbox, select the same tools you usually use. Here, I’m using the following Dockerfile and Docker to create a base image.
FROM ubuntu:focalSHELL ["/bin/bash", "-c"]# Install baseline packages
RUN apt-get update && \
DEBIAN_FRONTEND="noninteractive" apt-get install --yes \
apt-transport-https \
bash \
build-essential \
ca-certificates \
curl \
dbus \
dnsutils \
docker.io \
gnupg-agent \
htop \
iptables \
iproute2 \
iputils-ping \
jq \
kmod \
libsystemd0 \
locales \
lsb-release \
man \
openssh-server \
python3 \
python3-pip \
rsyslog \
software-properties-common \
sudo \
systemd \
systemd-sysv \
udev \
unzip \
vim \
wget && \
# Install Docker using official repository
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add — && \
apt-key fingerprint 0EBFCD88 && \
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" && \
apt-get update && apt-get install --no-install-recommends --yes \
docker-ce \
docker-ce-cli \
containerd.io && \
# Install latest Git using their official PPA
add-apt-repository ppa:git-core/ppa && \
DEBIAN_FRONTEND="noninteractive" apt-get install --yes git && \
# Housekeeping
apt-get autoremove --yes && \
apt-get clean --yes && \
rm -rf \
/tmp/* \
/var/lib/apt/lists/* \
/var/tmp/*# Prevents journald from reading kernel messages from /dev/kmsg
RUN echo "ReadKMsg=no" >> /etc/systemd/journald.conf# Create default ‘admin/admin’ user
RUN useradd admin \
--create-home \
--shell /bin/bash \
--groups docker \
--uid=1000 \
--user-group && \
echo "admin:admin" | chpasswd && \
echo "admin ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers.d/admin && \
mkdir /home/admin/.ssh && \
chown admin:admin /home/admin/.ssh# Make use of stopsignal (instead of sigterm) to stop systemd containers.
STOPSIGNAL SIGRTMIN+3# Expose OpenSSH server
EXPOSE 22# Set systemd as entrypoint.
ENTRYPOINT [ "/sbin/init", " — log-level=err" ]
Execute the docker build command to build the sysbox-base image, then push it to your container registry (identified by <registry> in the following commands).
$ docker build -t sysbox-base .
$ docker tag sysbox-base <registry>/sysbox-base:latest
$ docker push <registry>/sysbox-base:latestSysbox on Docker host
Installation
From a Docker host, the Sysbox package is installed with the apt install command:$ sudo apt install ./sysbox-ce_0.4.1-0.ubuntu-focal_amd64.deb
See the complete installation guide for Docker host on the following page: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-package.md#available-sysbox-packages
Docker-in-Docker
Now, you’re ready to run your first Sysbox container on Docker.
- To run the sysbox-base image, just built above, execute the docker run command with the runtime parameter.
$ docker run --runtime=sysbox-runc --hostname=sysbox -it sysbox-base - A regular VM startup displays on your terminal, followed by a login prompt.
- Login with admin username and admin password
- Check systemd services by executing the following command
admin@sysbox$ systemctl status - Execute a docker run command inside the Sysbox container
admin@sysbox$ docker run hello-world - Now, shut down the “container VM.”
admin@sysbox$ sudo shutdown now
Sysbox on Kubernetes Cluster
Installation
To install Sysbox on Kubernetes, follow these steps:
- To prepare the Sysbox installation on a Kubernetes cluster, add the sysbox-install=yes label to the worker nodes
$ kubectl label nodes <node> sysbox-install=yes - Apply the required manifests to add the required RBAC, install Sysbox on each worker node, and add sysbox-runc runtime.
$ kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox/master/sysbox-k8s-manifests/rbac/sysbox-deploy-rbac.yaml
$ kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox/master/sysbox-k8s-manifests/daemonset/sysbox-deploy-k8s.yaml
$ kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox/master/sysbox-k8s-manifests/runtime-class/sysbox-runtimeclass.yamlSee the complete installation guide for the Kubernetes cluster on the following page: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-k8s.md
Docker-in-Kubernetes
Now, you’re ready to run your first Sysbox container on Kubernetes.
- First, create a Pod manifest using sysbox-runc as runtimeClassName, and annotation to indicate to CRI-O to auto-assign user-namespace mappings, see the manifest content below.
- Then, apply this manifest to launch a related Pod on the Kubernetes cluster.
$ kubectl apply -f sysbox.yaml - Check the sysbox Pod is running well with the following command
$ kubectl get pods sysbox - Expose the SSH port as NodePort
$ kubectl expose pod sysbox --port=22 --name=sysbox --type=NodePort - Connect with SSH using the exposed NodePort (replace <node> with a node name.
$ ssh admin@<node> -p $(kubectl get service sysbox -o jsonpath="{.spec.ports[0].nodePort}") - Check systemd services by executing the following command
admin@sysbox$ systemctl status - Execute a docker run command inside the Sysbox container
admin@sysbox$ docker run hello-world - Now, shut down the “container VM.”
admin@sysbox$ sudo shutdown now
apiVersion: v1
kind: Pod
metadata:
name: sysbox
annotations:
io.kubernetes.cri-o.userns-mode: "auto:size=65536"
labels:
run: sysbox
spec:
runtimeClassName: sysbox-runc
containers:
— name: sysbox
image: <registry>/sysbox-base
ports:
- containerPort: 22
restartPolicy: Never