Kubernetes-in-Docker with kubeadm and Sysbox
Using Docker containers as Kubernetes nodes (aka KinD)
This article presents a walkthrough to create Kubernetes clusters using Docker containers acting as the Kubernetes nodes with Sysbox as the container runtime. The Linux host is installed with Ubuntu 20.04.
To know more about Sysbox, please read the What’s Sysbox by Nestybox? article.
As a result, the Kubernetes cluster is a set of Docker containers powered by Sysbox container runtime: one master node and two worker nodes.
Inside the Sysbox-powered containers, the inner containers are managed by containerd.
Container Image
The following image uses the sysbox-base image as a base (described in the What’s Sysbox by Nestybox? article).
Then, as the inner containers are managed by containerd, Docker service and socket could be disabled.
Next, kubeadm and kubelet packages are installed.
Finally, the required images are pre-pulled (with k8s-pull.sh script) into the built image to speed up the Kubernetes cluster creation.
Dockerfile
FROM sysbox-baseARG k8s_version=v1.20.12
ARG flannel_version=v0.15.1# Disable Docker
RUN systemctl disable docker.service docker.socket \
&& rm -f /etc/containerd/config.toml# Install kubeadm, kubelet
RUN curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add \
&& apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main" \
&& apt-get update && apt-get install --no-install-recommends -y \
kubeadm="${k8s_version#v}"-00 kubelet="${k8s_version#v}"-00 \
&& rm -rf /var/lib/apt/lists/*# Pre-pull Kubernetes images
COPY k8s-pull.sh /usr/bin/
RUN chmod +x /usr/bin/k8s-pull.sh && k8s-pull.sh $k8s_version $flannel_version && rm /usr/bin/k8s-pull.sh
k8s-pull.sh
#!/usr/bin/shk8s_version=$1
flannel_version=$2# Start containerd
containerd &
containerd_pid=$!
sleep 2# Pull k8s images
kubeadm config images list --kubernetes-version=$k8s_version | xargs -n 1 ctr images pull# Pull Flannel CNI images
ctr images pull docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.0
ctr images pull quay.io/coreos/flannel:$flannel_version# List pulled images
ctr images list# Stop containerd
kill $containerd_pid
Thus, you build the sysbox-k8s-node image with the regular docker build command: docker build -t sysbox-k8s-node . .
kind-create script
The following bash script (named kind-create) creates three Docker containers using the sysbox-k8s-node image and sysbox-runc container.
sysbox.kubeconfig file is copied from master node to local machine for interacting with KubeAPI.
#!/usr/bin/env bash
set -x
set -ek8s_version=v1.20.12
k8s_node_image=sysbox-k8s-node
k8s_network=sysbox
k8s_pod_cidr=10.244.0.0/16
flannel_version=v0.15.1# Create Kubernetes nodes
docker network create $k8s_network
docker run --runtime=sysbox-runc -d \
--name sysbox-master --hostname sysbox-master \
--net $k8s_network \
$k8s_node_image
docker run --runtime=sysbox-runc -d \
--name sysbox-worker-1 --hostname sysbox-worker-1 \
--net $k8s_network \
$k8s_node_image
docker run --runtime=sysbox-runc -d \
--name sysbox-worker-2 --hostname sysbox-worker-2 \
--net $k8s_network \
$k8s_node_image# Initialise the master node
docker exec sysbox-master sh -c \
"kubeadm init --kubernetes-version=$k8s_version --cri-socket=/var/run/containerd/containerd.sock --pod-network-cidr=$k8s_pod_cidr"# Download kubeconfig
docker cp sysbox-master:/etc/kubernetes/admin.conf sysbox.kubeconfig# Configure Flannel CNI
kubectl --kubeconfig=sysbox.kubeconfig apply -f https://raw.githubusercontent.com/coreos/flannel/$flannel_version/Documentation/kube-flannel.yml# Verify master node is good
kubectl --kubeconfig=sysbox.kubeconfig get all -A# Join the worker nodes
join_cmd=$(docker exec sysbox-master sh -c \
"kubeadm token create --print-join-command 2> /dev/null")
docker exec sysbox-worker-1 sh -c "$join_cmd"
docker exec sysbox-worker-2 sh -c "$join_cmd"# Verify all is good
kubectl --kubeconfig=sysbox.kubeconfig get all -A
kind-delete script
The following bash script (named kind-delete) deletes the Kubernetes nodes.
#!/usr/bin/env bash
set -x# Delete Kubernetes nodes
docker stop sysbox-master sysbox-worker-1 sysbox-worker-2
docker rm sysbox-master sysbox-worker-1 sysbox-worker-2
docker network rm sysbox
Create the Kubernetes cluster
- Execute kind-create script
- Verify Kubernetes nodes (i.e. docker containers) are running
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c545f158c196 sysbox-k8s-node “/sbin/init” 3 minutes ago Up 3 minutes 22/tcp sysbox-worker-2
45e8bcafc7bd sysbox-k8s-node “/sbin/init” 3 minutes ago Up 3 minutes 22/tcp sysbox-worker-1
9f692be9dd2b sysbox-k8s-node “/sbin/init” 3 minutes ago Up 3 minutes 22/tcp sysbox-master- Verify Kubernetes nodes are running with kubectl
$ kubectl --kubeconfig=sysbox.kubeconfig get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
sysbox-master Ready control-plane,master 7m45s v1.20.12 172.254.224.2 <none> Ubuntu 20.04.3 LTS 5.4.0-77-generic containerd://1.4.12
sysbox-worker-1 Ready <none> 7m18s v1.20.12 172.254.224.3 <none> Ubuntu 20.04.3 LTS 5.4.0-77-generic containerd://1.4.12
sysbox-worker-2 Ready <none> 7m9s v1.20.12 172.254.224.4 <none> Ubuntu 20.04.3 LTS 5.4.0-77-generic containerd://1.4.12Use the Kubernetes cluster
For example, run the NGINX image and verify it is running.
$ kubectl --kubeconfig=sysbox.kubeconfig run nginx --image=nginx
pod/nginx created
$ kubectl --kubeconfig=sysbox.kubeconfig get pods
NAME READY STATUS RESTARTS AGE
nginx 1/1 Running 0 10sDelete the Kubernetes cluster
- Execute kind-delete script
