Open in app

Sign in

Write

Sign in

Home

Library

Stories

Stats

Using OCI Volume Source in Kubernetes Pods

Stéphane Este-Gracias

--

Kubernetes 1.31 introduces an exciting new feature: the OCI Volume Source, currently in alpha. This feature allows mounting container images directly as read-only volumes in Kubernetes Pods, simplifying how applications can access static data resources such ML models or data patterns to share across your Kubernetes clusters.

This feature allows for easy static data distribution without having to develop a specific delivery service. OCI standards already specify everything for the image format and the use of an OCI registry for its distribution.

In this hands-on guide, we’ll explore how to set up and work with OCI Volume Source. Before diving into the hands-on section, let’s first explore how this feature originated and how it has been implemented.

KEP — Kubernetes Enhancement Proposal

The OCI Volume Source feature was proposed, discussed, and selected for implementation using a KEP (Kubernetes Enhancement Proposal), namely KEP-4639 supported by the SIG Node and the SIG Storage.

In Kubernetes, features follow a well-defined lifecycle — see Kubernetes — Features stages. The feature often starts from an idea from a developer, initially taking shape through online discussions. This raw idea typically matures into a formal Kubernetes Enhancement Proposal (KEP), which establishes the foundation for the feature’s development.

The KEP number corresponds to an issue number on a Kubernetes Git repository, namelykubernetes/enhancements, to track the lifecycle of this proposal, and the associated motivation is documented in the folder corresponding to the related responsible SIG on the same Git repository.

In a KEP, you have to define a summary that gives a brief overview of the enhancement, explaining its purpose. Then, the motivation section outlines why the feature is needed and its impact on Kubernetes. Goals define the intended outcomes, while non-goals clarify what the proposal won’t cover. Then, the proposal section provides a detailed technical description. It includes any API changes, design specifics, and user stories to illustrate use cases. Finally, it discusses potential risks and how they might be mitigated.

For instance, in this KEP document, you can find the planned roadmap for this feature, namely, it has just been released in alpha for Kubernetes 1.31, is scheduled for beta in version 1.32, and for stable in version 1.33.

OCI Volume Source Implementation

The OCI Volume Source feature defines a new interface for the Container Runtime Interface (CRI) that container runtimes such as containerd or CRI-O must implement.

The implementation on the Kubernetes side has been done and can be enabled using a feature flag, because it still in alpha mode, so it’s not enabled by default. To enable it, an option must be passed to the featureGates parameter for the API server and for the Kubelet on each node.

After a POD is scheduled on a specific node, when starting a POD with an OCI Image Volume specified, the Kubelet informs the container runtime through the CRI API to pull the image from the registry and mount it to the container.

Kubernetes Container Runtime Interface (CRI) and container image ecosystem

The container runtimes already implement the image pulling and its local storage on the node (used by the container image), so this functionality is already present on containerd and CRI-O. On the other hand, the new aspect of mounting the volume into a container needs to be implemented in the container runtime.

Installation

I’m used to using kind in my experiments, so I’ll use kind for this experiment and I’ll use the ORAS tool to create an artifact image that should be mounted in the container.

For information, kind is using kubeadm to deploy Kubernetes components into Docker containers using containerd as container runtime.

Here are the steps to follow:

  • Install Kind
$ curl -Lo ./kind <https://kind.sigs.k8s.io/dl/v0.24.0/kind-linux-amd64>
$ chmod +x ./kind
$ sudo mv ./kind /usr/local/bin
$ kind version
kind v0.24.0 go1.22.5 linux/amd64
  • Create YAML file cluster.yaml to use Kubernetes 1.31 and enable the ImageVolume feature.
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  ImageVolume: true
nodes:
  - role: control-plane
    image: kindest/node:v1.31.0@sha256:53df588e04085fd41ae12de0c3fe4c72f7013bba32a20e7325357a1ac94ba865
  • Create the cluster
$ kind create cluster --config cluster.yaml
  • Verify the nodes and the containerd version, kind 0.24 is using containerd 1.7.18
NAME                 STATUS   ROLES           AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION       CONTAINER-RUNTIME
kind-control-plane   Ready    control-plane   7m18s   v1.31.0   172.25.0.8    <none>        Debian GNU/Linux 12 (bookworm)   5.15.0-117-generic   containerd://1.7.18
  • Verify the feature flag is set on Kube API server
$ kubectl -n kube-system describe pod kube-apiserver-kind-control-plane
...
   Command:
      kube-apiserver
      ...
      --feature-gates=ImageVolume=true
      ...
...
  • Verify the features flag is set on Kubelet
$ docker exec kind-control-plane cat /var/lib/kubelet/config.yaml
...
featureGates:
  ImageVolume: true
...

So, the test cluster should be ready to test the OCI Volume Source feature.

  • Create an image artifact using ORAS command and push it to an OCI registry
$ echo "Hello ImageVolume" > artifact.txt
$ IMAGE_NAME=$(uuidgen)
$ oras push ttl.sh/${IMAGE_NAME}:1h \\
    --artifact-type application/vnd.acme.config \\
    artifact.txt:text/plain

Usage

  • Create a POD with a defined image volume
apiVersion: v1
kind: Pod
metadata:
  name: pod-oci-volume-source
spec:
  containers:
    - name: test
      image: nginx
      volumeMounts:
        - name: volume
          mountPath: /mnt
  volumes:
    - name: volume
      image:
        reference: ttl.sh/${IMAGE_NAME}:1h
        pullPolicy: IfNotPresent
  • Check container image (nginx) and image artifact (ttl.sh/xxx:1h) have been pulled locally on the Kubernetes node by the container runtime using crictl command.
$ docker exec kind-control-plane crictl image
IMAGE                                           TAG                  IMAGE ID            SIZE
docker.io/kindest/kindnetd                      v20240813-c6f155d6   12968670680f4       36.8MB
docker.io/kindest/local-path-helper             v20230510-486859a6   be300acfc8622       3.05MB
docker.io/kindest/local-path-provisioner        v20240813-c6f155d6   3a195b56ff154       19.4MB
docker.io/library/nginx                         latest               5ef79149e0ec8       71MB
registry.k8s.io/coredns/coredns                 v1.11.1              cbb01a7bd410d       18.2MB
registry.k8s.io/etcd                            3.5.15-0             2e96e5913fc06       56.9MB
registry.k8s.io/kube-apiserver-amd64            v1.31.0              4f8c99889f8e4       95.2MB
registry.k8s.io/kube-apiserver                  v1.31.0              4f8c99889f8e4       95.2MB
registry.k8s.io/kube-controller-manager-amd64   v1.31.0              7e9a7dc204d9d       89.4MB
registry.k8s.io/kube-controller-manager         v1.31.0              7e9a7dc204d9d       89.4MB
registry.k8s.io/kube-proxy-amd64                v1.31.0              af3ec60a3d89b       92.7MB
registry.k8s.io/kube-proxy                      v1.31.0              af3ec60a3d89b       92.7MB
registry.k8s.io/kube-scheduler-amd64            v1.31.0              418e326664bd2       68.4MB
registry.k8s.io/kube-scheduler                  v1.31.0              418e326664bd2       68.4MB
registry.k8s.io/pause                           3.10                 873ed75102791       320kB
ttl.sh/fab04c8a-6d9d-4c9a-b2d6-568b0193dfe0     1h                   44136fa355b36       575B

But, the POD is not running and we get some weird error on the events.

$ kubectl get pods
NAME                    READY   STATUS             RESTARTS   AGE
pod-oci-volume-source   0/1     ImagePullBackOff   0          3m
Warning  Failed     3s    kubelet            Failed to pull image "ttl.sh/fab04c8a-6d9d-4c9a-b2d6-568b0193dfe0:1h": failed to pull and unpack image "ttl.sh/fab04c8a-6d9d-4c9a-b2d6-568b0193dfe0:1h": number of layers and diffIDs don't match: 1 != 0

After asking to the community, I got more insights. Thanks of them. Because, containerd 1.7.18 doesn’t implement the new CRI API required for the Image Volume feature yet.

To implement this new feature, the container runtime projects (CRI-O and containerd) use both a respective PR in their respective Git repository to follow the implementation of this feature.

As this writing, none of them released an official version of the container runtime yet.

So, let’s wait for their implementation and I will update this article later.

Conclusion

The OCI Volume Source feature allows to mount container images directly as read-only volumes in Kubernetes Pods, simplifying how applications can access static data resources such ML models or data patterns to share across you Kubernetes clusters.

To access this functionality on Kubernetes 1.31, it is necessary to enable the ImageVolume feature gate, as well as use a container runtime that implements the new CRI API required for this functionality to work.

This functionality is promising and will certainly simplify the needs for sharing static data such as ML models or any other data patterns that need to be distributed across your different clusters.

To be continued…

References

Sign up to discover human stories that deepen your understanding of the world.

--

--

Stéphane Este-Gracias
Stéphane Este-Gracias

Written by Stéphane Este-Gracias

54 Followers
21 Following

Open Source & Cloud Native Innovation Lead @ ITQ

No responses yet

Write a response

What are your thoughts?