Kubernetes Control Plane Scaling: 3 Limits, Not One Node Count

The standard fix for a struggling Kubernetes control plane, adding more apiservers, can make it worse, because every apiserver replicates the entire watch cache: you buy request capacity and pay for it in memory you didn’t have. For platform teams running AI training fleets or agent platforms, a New Stack piece argues that the problem starts earlier, with measuring scale as a single node count.

The Framework

The New Stack article describes three clusters in trouble that look nothing alike, and treats them as three independent limits. Any one can be maxed out while the other two stay low, so a recipe tuned only for node count can tune the wrong thing.

Node-bound. Large training and reinforcement-learning fleets, where the pressure is every kubelet reporting in. At 100,000 nodes with a 10-second heartbeat, liveness alone generates about 10,000 writes per second, and in the authors’ measurements, lease renewals were roughly 70% of write traffic.

Churn-bound. Agent platforms of modest size, where each agent session is a custom resource that is created, suspended, resumed and destroyed thousands of times a minute. The write rate has nothing to do with machine count.

State-bound. Platform clusters with hundreds of operators and tens of thousands of mostly idle objects. The cost is resident memory multiplied by every apiserver holding a copy, which is why adding apiservers hurts here most.

The Four Seams

The authors call the fixes seams, contracts between components that can be changed without patching upstream.

The storage contract relieves churn. Events and leases are ephemeral and don’t need the same Raft-replicated store as Deployments, and the --etcd-servers-overrides flag can route resource types to different backends today. KEP-4743 defines the storage-interface boundary. The gap: the override only knows built-in resources, so the custom resources driving an agent platform’s write storm can’t be routed yet.

The node liveness contract relieves node count, either with a node-facing tier that absorbs lease renewals or by routing renewals through etcd’s in-memory lease refresh instead of a full Raft commit, a change that would help clusters of any size.

The shard axis relieves all three. KEP-5866 introduces hash-range filtering for API traffic, though coordination, resharding and sharding the controller manager are still out of scope.

Cache residency relieves state by letting an apiserver keep only the objects it needs; a two-shard prototype cut heap by roughly half per shard. Residency doesn’t exist upstream yet.

Where It Doesn’t Apply

The authors publish a ladder of recipes on stock upstream components: classic high availability up to 5,000 nodes, role-split control planes from 5,000 to 20,000, a node plane tier to 65,000, partitioned stores around 100,000, and a sharded design beyond that which “doesn’t exist yet.” Their own caveat is the important one: the ladder is indexed by node count, and no one has published a scale-tested recipe for a churn-bound or state-bound cluster. For now, agent platforms have no published recipe to follow. Stack Archive covered a related operational lesson in EKS Kubernetes version rollbacks.

Source: The New Stack

Content created with AI assistance and reviewed for accuracy.