Using OCI Volume Source in Kubernetes Pods

Stéphane Este-Gracias
6 min readAug 21, 2024

--

Kubernetes 1.31 introduces an exciting new feature: the OCI Volume Source, currently in alpha. This feature allows mounting container images directly as read-only volumes in Kubernetes Pods, simplifying how applications can access static data resources such ML models or data patterns to share across your Kubernetes clusters.

This feature allows for easy static data distribution without having to develop a specific delivery service. OCI standards already specify everything for the image format and the use of an OCI registry for its distribution.

In this hands-on guide, we’ll explore how to set up and work with OCI Volume Source. Before diving into the hands-on section, let’s first explore how this feature originated and how it has been implemented.

KEP — Kubernetes Enhancement Proposal

The OCI Volume Source feature was proposed, discussed, and selected for implementation using a KEP (Kubernetes Enhancement Proposal), namely KEP-4639 supported by the SIG Node and the SIG Storage.

In Kubernetes, features follow a well-defined lifecycle — see Kubernetes — Features stages. The feature often starts from an idea from a developer, initially taking shape through online discussions. This raw idea typically matures into a formal Kubernetes Enhancement Proposal (KEP), which establishes the foundation for the feature’s development.

The KEP number corresponds to an issue number on a Kubernetes Git repository, namelykubernetes/enhancements, to track the lifecycle of this proposal, and the associated motivation is documented in the folder corresponding to the related responsible SIG on the same Git repository.

In a KEP, you have to define a summary that gives a brief overview of the enhancement, explaining its purpose. Then, the motivation section outlines why the feature is needed and its impact on Kubernetes. Goals define the intended outcomes, while non-goals clarify what the proposal won’t cover. Then, the proposal section provides a detailed technical description. It includes any API changes, design specifics, and user stories to illustrate use cases. Finally, it discusses potential risks and how they might be mitigated.

For instance, in this KEP document, you can find the planned roadmap for this feature, namely, it has just been released in alpha for Kubernetes 1.31, is scheduled for beta in version 1.32, and for stable in version 1.33.

OCI Volume Source Implementation

The OCI Volume Source feature defines a new interface for the Container Runtime Interface (CRI) that container runtimes such as containerd or CRI-O must implement.

The implementation on the Kubernetes side has been done and can be enabled using a feature flag, because it still in alpha mode, so it’s not enabled by default. To enable it, an option must be passed to the featureGates parameter for the API server and for the Kubelet on each node.

After a POD is scheduled on a specific node, when starting a POD with an OCI Image Volume specified, the Kubelet informs the container runtime through the CRI API to pull the image from the registry and mount it to the container.

Kubernetes Container Runtime Interface (CRI) and container image ecosystem

The container runtimes already implement the image pulling and its local storage on the node (used by the container image), so this functionality is already present on containerd and CRI-O. On the other hand, the new aspect of mounting the volume into a container needs to be implemented in the container runtime.

Installation

I’m used to using kind in my experiments, so I’ll use kind for this experiment and I’ll use the ORAS tool to create an artifact image that should be mounted in the container.

For information, kind is using kubeadm to deploy Kubernetes components into Docker containers using containerd as container runtime.

Here are the steps to follow:

  • Install Kind
$ curl -Lo ./kind <https://kind.sigs.k8s.io/dl/v0.24.0/kind-linux-amd64>
$ chmod +x ./kind
$ sudo mv ./kind /usr/local/bin
$ kind version
kind v0.24.0 go1.22.5 linux/amd64
  • Create YAML file cluster.yaml to use Kubernetes 1.31 and enable the ImageVolume feature.
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
ImageVolume: true
nodes:
- role: control-plane
image: kindest/node:v1.31.0@sha256:53df588e04085fd41ae12de0c3fe4c72f7013bba32a20e7325357a1ac94ba865
  • Create the cluster
$ kind create cluster --config cluster.yaml
  • Verify the nodes and the containerd version, kind 0.24 is using containerd 1.7.18
NAME                 STATUS   ROLES           AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION       CONTAINER-RUNTIME
kind-control-plane Ready control-plane 7m18s v1.31.0 172.25.0.8 <none> Debian GNU/Linux 12 (bookworm) 5.15.0-117-generic containerd://1.7.18
  • Verify the feature flag is set on Kube API server
$ kubectl -n kube-system describe pod kube-apiserver-kind-control-plane
...
Command:
kube-apiserver
...
--feature-gates=ImageVolume=true
...
...
  • Verify the features flag is set on Kubelet
$ docker exec kind-control-plane cat /var/lib/kubelet/config.yaml
...
featureGates:
ImageVolume: true
...

So, the test cluster should be ready to test the OCI Volume Source feature.

  • Create an image artifact using ORAS command and push it to an OCI registry
$ echo "Hello ImageVolume" > artifact.txt
$ IMAGE_NAME=$(uuidgen)
$ oras push ttl.sh/${IMAGE_NAME}:1h \\
--artifact-type application/vnd.acme.config \\
artifact.txt:text/plain

Usage

  • Create a POD with a defined image volume
apiVersion: v1
kind: Pod
metadata:
name: pod-oci-volume-source
spec:
containers:
- name: test
image: nginx
volumeMounts:
- name: volume
mountPath: /mnt
volumes:
- name: volume
image:
reference: ttl.sh/${IMAGE_NAME}:1h
pullPolicy: IfNotPresent
  • Check container image (nginx) and image artifact (ttl.sh/xxx:1h) have been pulled locally on the Kubernetes node by the container runtime using crictl command.
$ docker exec kind-control-plane crictl image
IMAGE TAG IMAGE ID SIZE
docker.io/kindest/kindnetd v20240813-c6f155d6 12968670680f4 36.8MB
docker.io/kindest/local-path-helper v20230510-486859a6 be300acfc8622 3.05MB
docker.io/kindest/local-path-provisioner v20240813-c6f155d6 3a195b56ff154 19.4MB
docker.io/library/nginx latest 5ef79149e0ec8 71MB
registry.k8s.io/coredns/coredns v1.11.1 cbb01a7bd410d 18.2MB
registry.k8s.io/etcd 3.5.15-0 2e96e5913fc06 56.9MB
registry.k8s.io/kube-apiserver-amd64 v1.31.0 4f8c99889f8e4 95.2MB
registry.k8s.io/kube-apiserver v1.31.0 4f8c99889f8e4 95.2MB
registry.k8s.io/kube-controller-manager-amd64 v1.31.0 7e9a7dc204d9d 89.4MB
registry.k8s.io/kube-controller-manager v1.31.0 7e9a7dc204d9d 89.4MB
registry.k8s.io/kube-proxy-amd64 v1.31.0 af3ec60a3d89b 92.7MB
registry.k8s.io/kube-proxy v1.31.0 af3ec60a3d89b 92.7MB
registry.k8s.io/kube-scheduler-amd64 v1.31.0 418e326664bd2 68.4MB
registry.k8s.io/kube-scheduler v1.31.0 418e326664bd2 68.4MB
registry.k8s.io/pause 3.10 873ed75102791 320kB
ttl.sh/fab04c8a-6d9d-4c9a-b2d6-568b0193dfe0 1h 44136fa355b36 575B

But, the POD is not running and we get some weird error on the events.

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pod-oci-volume-source 0/1 ImagePullBackOff 0 3m
Warning  Failed     3s    kubelet            Failed to pull image "ttl.sh/fab04c8a-6d9d-4c9a-b2d6-568b0193dfe0:1h": failed to pull and unpack image "ttl.sh/fab04c8a-6d9d-4c9a-b2d6-568b0193dfe0:1h": number of layers and diffIDs don't match: 1 != 0

After asking to the community, I got more insights. Thanks of them. Because, containerd 1.7.18 doesn’t implement the new CRI API required for the Image Volume feature yet.

To implement this new feature, the container runtime projects (CRI-O and containerd) use both a respective PR in their respective Git repository to follow the implementation of this feature.

As this writing, none of them released an official version of the container runtime yet.

So, let’s wait for their implementation and I will update this article later.

Conclusion

The OCI Volume Source feature allows to mount container images directly as read-only volumes in Kubernetes Pods, simplifying how applications can access static data resources such ML models or data patterns to share across you Kubernetes clusters.

To access this functionality on Kubernetes 1.31, it is necessary to enable the ImageVolume feature gate, as well as use a container runtime that implements the new CRI API required for this functionality to work.

This functionality is promising and will certainly simplify the needs for sharing static data such as ML models or any other data patterns that need to be distributed across your different clusters.

To be continued…

References

Sign up to discover human stories that deepen your understanding of the world.

Free

Distraction-free reading. No ads.

Organize your knowledge with lists and highlights.

Tell your story. Find your audience.

Membership

Read member-only stories

Support writers you read most

Earn money for your writing

Listen to audio narrations

Read offline with the Medium app

--

--

Stéphane Este-Gracias
Stéphane Este-Gracias

Written by Stéphane Este-Gracias

Open Source & Cloud Native Innovation Lead @ ITQ

No responses yet

Write a response