Eleven Minutes, Zero Humans: How This Kubernetes Upgrade Pipeline Works

Most Kubernetes control plane upgrades require a human. They require someone to SSH in, watch etcd health, and verify node rejoin after each reboot. A July production run on Kairos completed the same operation — three control-plane nodes, one at a time — in eleven minutes with a two-line Git commit as the only human input.

What Kairos Is

Kairos is an immutable Linux distribution designed around A/B partition upgrades. Rather than patching packages in place, it writes a new OS image to an inactive partition and reboots into it. The previous partition remains intact. Rolling back is a reboot — not a recovery procedure. This design makes the upgrade unit the entire OS image, not individual packages, which eliminates the drift that accumulates in traditional Kubernetes node maintenance.

The Six-Tool Pipeline

The eleven-minute result requires six tools working in sequence, each with a specific role:

1. Gitea — self-hosted Git repository with CI runners. The source of truth for node configuration.

2. Renovate — automated dependency bumping. When Kairos releases a new version, Renovate opens a PR against the Gitea repo within its scanning window. No human monitors release notes.

3. Kyverno — ClusterPolicy admission controller. Blocks misconfigurations before they reach the cluster. Acts as the gate between “a PR was merged” and “the cluster is modified.”

4. Cosign — cryptographic image signature verification. Before any image is deployed, Cosign verifies it against the upstream CI identity. This is the supply chain security layer — it ensures that the image being deployed originated from the Kairos build pipeline, not a shadow build.

5. ArgoCD — GitOps deployment. Detects the merged version bump as drift between the desired state (Git) and the actual state (cluster) and triggers reconciliation.

6. kairos-operator — custom resource executor. Handles node cordoning, sequential upgrade execution, and post-reboot rejoin validation. This is the component that actually does the work; the others provide the governance and delivery layers above it.

The Critical Concurrency Setting

The near-miss in this pipeline is worth understanding in detail. The concurrency setting in the kairos-operator controls how many nodes upgrade simultaneously. Setting concurrency: 0 upgrades all nodes at once — which means all three control-plane nodes reboot simultaneously. etcd quorum requires a majority of nodes to be healthy; losing all three simultaneously breaks quorum and potentially corrupts cluster state.

The production pipeline had concurrency: 0 initially. Per the CNCF writeup, it “only survived by chance rather than design.” The correct setting is concurrency: 1 — one node at a time, with rejoin validation before the next node begins. This adds roughly two minutes per node but is the difference between a reliable pipeline and a high-stakes gamble.

The per-node timeline with concurrency: 1: image pull takes 25.5 seconds for a 324 MB image; the upgrade and reboot cycle runs approximately 2 minutes per node. Three nodes, sequentially: approximately 7–8 minutes of node operations plus coordination overhead equals the 11-minute total.

Source: Cloud Native Computing Foundation

Content created with AI assistance and reviewed for accuracy.