Kubernetes Autoscaling for AI: Why GPU Pods Stay Pending

A GPU pod sits pending, the team assumes the region is out of accelerators, and the capacity is actually there: the node the autoscaler is willing to add is simply not one the scheduler is willing to use. That is the failure two AWS engineers describe on the CNCF blog, and for platform teams paying for scarce accelerators it means idle, expensive hardware caused by two Kubernetes control loops that rarely had to talk to each other.

How It Works

Writing on the CNCF blog, AWS’s Sajjan Gundapuneedi and Neelendra Bhandari explain the split. The scheduler places each pod on a node that already exists; the cluster autoscaler adds nodes when pods have nowhere to fit. For a decade that division worked because compute was largely interchangeable. Accelerators changed that, and the authors identify three problems both loops now need to understand the same way.

Reservations. Teams reserve scarce accelerator capacity ahead of demand, so a scale-up decision has to consider whether a new node draws on capacity already committed and paid for, not just whether to grow.

Topology. Accelerators run best when the CPU, memory, network interface and peer accelerators they depend on sit close together, a hierarchy that runs from a single server to racks joined by a fast fabric. Kubernetes historically represented accelerators as bare counts. Dynamic Resource Allocation replaces the count with a structured claim describing what a workload needs. An autoscaler without DRA support may provision a node that doesn’t satisfy that claim.

Sharing. One modern accelerator is too powerful to hand whole to a small job. Partitionable devices offer a device as smaller slices, and consumable capacity lets several workloads draw measured portions. The scheduler tracks those allocations directly; DRA-aware autoscalers are starting to use the same model, though support still varies.

Where the Two Loops Still Disagree

A shared hardware language lets the loops agree on what a node is. It doesn’t make them act together, and gang scheduling shows the gap. A large training job needs every pod placed at once or not at all. Through workload-aware scheduling, whose Workload and PodGroup APIs are beta upstream and disabled by default, the scheduler is learning to treat a group as one unit. The autoscaler was not part of that change. It still reasons pod by pod, so it can add a node or two for a few members of a gang while the job never starts.

The current bridge is the provisioning request, which lets a workload ask for a whole group’s capacity at once. With a configured admission check, Kueue reserves quota, creates the request, waits for capacity and then admits the workload, though atomic scale-up is best effort and depends on the cloud provider. The authors call this a bolt-on, and point to a public SIG-Autoscaling proposal for APIs that would let the scheduler see capacity that could be provisioned, express placement preferences across it and request it directly.

What This Means for You

The authors’ advice for a pending GPU pod is to stop assuming capacity is gone and check whether the scheduler and autoscaler agree on what a usable node looks like. Inference makes this urgent, they argue, because bursty serving and agentic workloads have no patience for a cold node. Stack Archive’s guide to predictive autoscaling for GPU workloads tackles the same cold-start delay from the demand side.

Source: CNCF Blog

Content created with AI assistance and reviewed for accuracy.