How to Migrate Kubernetes Nodes to cgroup v2 Before v1.38

Summary

• Starting with Kubernetes v1.35, the kubelet setting failCgroupV1 defaults to true, so the kubelet does not start on a cgroup v1 node unless you override it
• In kubeadm clusters, the v1.35 preflight check now fails kubeadm init, join, and upgrade on cgroup v1 nodes instead of warning
• The cgroup v1 fallback still works in v1.36 but is scheduled for removal in v1.38 (KEP-5573)
• cgroup v2 support has been stable since v1.25 and unlocks Memory QoS, PSI metrics, container-aware OOM handling, and accurate Pod-level resource enforcement
• Most migration risk sits outside Kubernetes: kernel versions, container runtimes, and agents that read the cgroup filesystem directly

Purpose

To move every Linux node in a cluster to cgroup v2 before an upgrade turns a deprecation warning into kubelets that will not start. The Kubernetes project’s guidance is direct: clusters below v1.35 should migrate every node before upgrading, and clusters already on v1.35 or later should confirm every node runs cgroup v2 or that the override is deliberate.

Pre-requisites

• Root access to each Linux node, or a node image pipeline that can rebuild them
• Linux kernel 5.8 or later (5.9 or later if you plan to enable Memory QoS)
• A cgroup v2-capable container runtime: containerd v1.4+ (v2.0+ for automatic cgroup driver discovery) or CRI-O v1.20+
• An inventory of DaemonSets and agents on nodes: monitoring, security, and service mesh components

Steps

1. Audit what every node runs today. On each node, run stat -fc %T /sys/fs/cgroup/. Output of cgroup2fs means cgroup v2. Output of tmpfs means cgroup v1, per the Kubernetes cgroup documentation. Record node pools, OS images, and kernel versions alongside the result. In managed clusters, check provider defaults per node type, because they can differ within one cluster.

2. Fix the kernel and OS image first. Kubernetes documents 5.8 as the minimum kernel for cgroup v2 and does not recommend anything older than 5.2. Memory QoS needs 5.9 or later to avoid a known memory.high livelock. Prefer rebuilding node images with cgroup v2 enabled over toggling hosts in place.

3. Align the cgroup driver. The kubelet and container runtime must use the same cgroup driver. For kubeadm-managed clusters, Kubernetes recommends the systemd driver. With containerd v2.0+ or CRI-O v1.28+, the kubelet detects the runtime’s driver automatically (stable since v1.34). On older runtimes, set cgroupDriver: systemd explicitly in the KubeletConfiguration.

4. Upgrade everything that reads cgroup files. This is where migrations break quietly. The guidance recommends cAdvisor v0.43.0 or later and lists minimum Java, Node.js, and automaxprocs versions. KEP-5573 cites OpenJDK jdk8u372, 11.0.16, or 15+, and Node.js 20.3.0+. Older runtimes can misread container memory and CPU limits under cgroup v2. Confirm third-party monitoring and security agents support v2 before rolling nodes.

5. Review the OOM behavior change. On cgroup v2, the kubelet defaults singleProcessOOMKill to false and sets memory.oom.group, so an OOM kills every process in the container together. For multi-process containers that relied on one worker dying while the rest kept serving, set singleProcessOOMKill: true only where that v1-style behavior is genuinely required.

6. Check CPU weight expectations. cgroup v1 uses cpu.shares and v2 uses cpu.weight. Newer runtimes (crun v1.23, runc v1.3.2) use an improved non-linear conversion. Policy or monitoring tools that predict exact cpu.weight values may need updating after a runtime upgrade.

7. Roll node pools and verify. Cordon, drain, and replace nodes pool by pool. On a migrated node, confirm limits landed by reading cpu.max and memory.max in the container’s cgroup directory, and use systemd-cgls or systemd-cgtop to inspect the hierarchy.

8. Treat the override as a dated exception. If one pool cannot move before a v1.35+ upgrade, set failCgroupV1: false in that pool’s kubelet configuration, record an owner and a removal date, and plan to finish before v1.38 removes the fallback.

The Trap

Assuming the migration is a node OS setting. The kubelet and runtime switch cleanly. The failures come from the long tail: an old JVM reporting the host’s memory instead of the container limit, or a monitoring agent that silently stops reporting per-container metrics. Staging the change on one node pool that runs a representative mix of DaemonSets catches these before production does. For a related look at kubelet-level resource controls that depend on cgroup v2, see Stack Archive’s guide to Kubernetes v1.36 Pod-level resource managers.

Cited References

• Kubernetes Blog: The Shift to cgroup v2 in Kubernetes: What You Need to Know
• Kubernetes Documentation: About cgroup v2
• KEP-5573: Remove cgroup v1 support

Source: Kubernetes Blog

Content created with AI assistance and reviewed for accuracy.