GitOps with Flux: Leveraging OCI Registry as a State Store
GitOps does not simply mean using Git with YAML configuration files and Kubernetes operators.
This article will explore what GitOps truly entails, its use cases, and available tools.
Furthermore, it will demonstrate how an OCI registry can be an alternative to a Git repository as a desired state store.
- Declarative: A system managed by GitOps must have its desired state expressed declaratively.
- Versioned and Immutable: Desired state is stored in a way that enforces immutability, versioning and retains a complete version history.
- Pulled Automatically: Software agents automatically pull the desired state declarations from the source.
- Continuously Reconciled: Software agents continuously observe actual system state and attempt to apply the desired state.
To better understand these important concepts, OpenGitOps has provided a list of terms to help for understanding the main ideas and details of each term. By using this list, these concepts can be applied in your work and can be explained to others more easily.
- Continuous: “Continuous” is intended to match the industry standard term: reconciliation continues to happen, not that it must be instantaneous.
- Declarative Description: A configuration that describes the desired operating state of a system without specifying procedures for how that state will be achieved. This separates configuration (the desired state) from the implementation (commands, API calls, scripts etc.) used to achieve that state.
- Desired State: The aggregate of all configuration data that is sufficient to recreate the system so that instances of the system are behaviourally indistinguishable. This configuration data generally does not include persistent application data, eg. database contents, though often does include credentials for accessing that data, or configuration for data recovery tools running on that system.
- Drift: When a system’s actual state has moved or is in the process of moving away from the desired state, this is often referred to as drift.
- Reconciliation: The process of ensuring the actual state of a system matches its desired state. Contrary to traditional CI/CD where automation is generally driven by pre-set triggers, in GitOps reconciliation is triggered whenever there is a divergence. Divergence could be due to the actual state unintentionally drifting from the desired state declarations, or a new desired state declaration version having been changed intentionally. Actions are taken based on policies around feedback from the system and previous reconciliation attempts, in order to reduce deviation over time.
- Software System: A software system managed by GitOps includes:
1. One or more runtime environments consisting of resources under management
2. The management agents within each runtime
3. Policies for controlling access and management of repositories, deployments, runtimes
- State Store: A system for storing immutable versions of desired state declarations. This state store should provide access control and auditing on the changes to the desired state. Git, from which GitOps derives its name, is the canonical example used as this state store but any other system that meets these criteria may be used. In all cases, these state stores must be properly configured and precautions must be taken to comply with requirements set out in the GitOps Principles.
- Feedback: Open GitOps follows control-theory and operates in a closed-loop. In control theory, feedback represents how previous attempts to apply a desired state have affected the actual state. For example if the desired state requires more resources than exist in a system, the software agent may make attempts to add resources, to automatically rollback to a previous version, or to send alerts to human operators.
Now that “What’s GitOps” was covered, let’s explore some of its use cases and take a closer look at some of the most common and impactful GitOps use cases.
GitOps Use Cases
The benefits of GitOps are numerous, including increased security, better auditability, and improved collaboration. In fact, GitOps has become so popular that it is now being used in a variety of different use cases such as the following.
- Infrastructure management: GitOps allows you to manage infrastructure as code (IaC), making it easy to version control, review, and apply infrastructure changes. This approach reduces human error.
- Application deployment: GitOps simplifies application deployment by automating the process and maintaining the desired state in a state store. This enables easy rollbacks, and version control.
- Multi-environment and multi-tenant management: GitOps makes it easy to manage multiple environments (e.g., dev, staging, prod) and multiple tenants with different configurations.
- Compliance and auditing: GitOps provides a transparent history of all changes, making it easy to track and audit infrastructure and application changes. This is particularly useful for organizations that need to adhere to strict compliance regulations.
- Disaster recovery: GitOps helps in disaster recovery by maintaining the desired state in a state store. In case of a disaster, you can easily restore the system to its last known good state by applying the configurations from the state store.
Leveraging the Platform Team through the implementation of these use cases can greatly enhance the effectiveness of GitOps.
Mainly, GitOps provides a streamlined approach to platform teams, enabling organizations to increase their efficiency. With the capability to centrally manage infrastructure and application deployment, developers can concentrate on their core responsibilities of software creation and improvement.
To implement GitOps and leverage its benefits, organizations need the right tools and technologies. Fortunately, there are many GitOps tools available that can help teams implement this approach quickly and effectively.
Before proceeding, let’s review the GitOps principles through tooling.
Just because something is declarative, it doesn’t mean that it must use YAML configuration files. It could also use other configuration files or software languages that compile, transform, or generate configurations. As long as the configuration can be stored and used as a source of trust in the right way, it can be considered declarative.
So, the possible solutions for defining a declarative configuration are:
- ytt (from Carvel projet)
- Cue Lang
- HCL (HCL is the HashiCorp configuration language)
- CDK for Terraform
Versioned and Immutable
Git’s versioning and immutability features are great, but not everyone enjoys the user experience using Git CLI. To address this, graphical user interfaces, automation, and other tools can be leveraged to improve the workflow. Possible UIs are the following:
- Git UI
- GitOps UI
- Argo CD web UI
- Weave GitOps web UI for Flux
- Rancher Fleet in Rancher UI
- Terraform Cloud web UI (using VCS workflow)
Then, most of GitOps tools rely on Git for the state store. In contrast, the Flux Source Controllers can use a Git repository, a storage bucket, or an OCI registry (see dedicated section below).
Pulled Automatically and Continuously Reconciled
GitOps tools use a periodic timer as a trigger for pulling configurations from the state store. This trigger can usually be paused, enabled or forced manually.
The reconciliation does not mean that it must necessarily use Kubernetes operators, but other reconciliations can be used: for instance, the drift detection for Terraform Cloud enables reconciliation using the Terraform state.
Combining GitOps principles, the following tools are possible GitOps toolsets with some useful extensions such as event notifications and deployment policy management:
- Argo CD
- Argo Rollouts
- Argo Events
- Flux Toolkit
- Source Controllers
- Kustomize Controllers
- Helm Controllers
- Notification Controller
- Image Automation Controller
- Terraform Controller
- Rancher Fleet
- Terraform Cloud
OCI registry as state store
OCI registry is a core component for the cloud native ecosystem.
More or more tools can relying on OCI registries to store their configurations. The following tools can leverage OCI registry as its source of trust.
- Config Sync by Google (from kpt project)
- Flux by CNCF (using OCIRepositories)
- Helm by CNCF (using OCI-based registries)
- imgpkg by CNCF (from Carvel project)
OCI registry can be used by Flux as a state store. The image tag immutability must be enforce to comply with the GitOps principles.
How to configure image tag immutability for the main registry providers.
- Amazon ECR (using image tag immutability)
- Azure Container Registry (using tag locking policy)
- Google Container Registry (feature request for image tag immutability)
- Harbor (using tag immutability rules)
Using an OCI registry as a state store for GitOps provides several advantages over using Git alone.
- Pulling an OCI image is a much less resource-intensive process compared to cloning a Git repository, making it more efficient for deployment purposes.
- High available registries are easier to deploy than Git servers, making it easy to access and deploy container images from any location.
- OCI repositories can scale out similarly to a Content Delivery Network, enabling platform team to distribute configurations globally with ease.
- Flux uses Kubernetes workload identity and IAM to pull OCI artifacts from managed registries. This eliminates the need for key management, SSH key generation, and proprietary API usage for token generation. Instead, the same mechanism used for pulling container images for applications is used.
- OCI images can be signed and verified during container deployment. Flux supports these verifications, out of the box.
Overall, leveraging Flux with the use of an OCI registry as a state store can enhance the scalability, reliability, and security of GitOps deployments.
Moreover, using OCI instead of Git can be particularly useful when the Git repository doesn’t contain the final configuration files. If you’re using ytt, CDK for Terraform, Pulumi, Cue Lang, Starlark, or any other tool that generates configuration files, the generators can be run in a build pipeline and the resulting configuration can be published as OCI artifacts using the Flux CLI for instance.
Flux GitOps Toolkit, in addition to its composability, fully relies on the standard Kubernetes API and can extract the desired state configuration from OCI registries, which is a significant advantage over ArgoCD or Rancher Fleet when targeting multiple Kubernetes clusters.
On the other hand, Terraform Cloud can be used to implement GitOps principles to manage everything as code, such as infrastructure, networking, applications codification…
- OpenGitOps — GitOps Principles
- OpenGitOps — GitOps Glossary
- HCL configuration files
- CDK for Terraform
- Cue Lang
- Flux GitOps Toolkit
- Rancher Fleet
- Terraform Cloud
- Drift Detection for Terraform Cloud
- Open Container Initiative
- Flux — OCI cheatsheet