AI Inference on Kubernetes: Why Busy GPUs Hide Slow Service

“Our GPUs are maxed at 100%. So we’re doing well.” HPE product manager Marius Bogoevici offers that line as a misleading victory lap: a Kubernetes cluster can be healthy and its GPUs fully busy while the AI service on top answers slowly or wrongly. For platform teams running inference, the job is connecting what users experience to the GPU, network and storage conditions behind it.

The Framework

The advice comes from an HPE-sponsored series in The New Stack, with Bogoevici and Karthik Subramanian of HPE Morpheus and Mahesh Ramachandran of HPE OpsRamp. HPE’s products appear in it, but the practices are vendor-neutral. Five steps stand out.

1. Measure the Experience, Not the Hardware

For a conversational service, Bogoevici lists time to first token, time per output token, reliability, capacity, resource efficiency, cost and answer quality. They answer different questions: “A quick first token can mask slow subsequent generation; a fast complete response can still be inaccurate or unhelpful.” Teams should record where each interval is measured and test under sustained load and spikes, with targets set per workload.

2. Profile Before Launch

Ramachandran’s rule is to “profile it before you launch.” Test representative prompt and context lengths, output volumes and traffic, then record token throughput, latency and service levels. That baseline lets operators tell rising demand apart from resource contention later. Answer quality needs its own tests and acceptance criteria from data science and domain experts, rerun after any model, prompt, retrieval or serving change.

3. Treat GPU Utilization as a Clue

When the service is slow and the GPU is busy, the speakers suggest checking whether inference pods scale on workload-appropriate metrics, whether requests reach the intended model, and whether related components can be scheduled. A change in request mix across models can change latency while GPU activity looks normal. Subramanian adds network configuration, database or vector-database response times, and whether “the GPU partition [is] too small to serve that model.” These are investigation paths, not a diagnosis for every cluster.

4. Scale on the Right Signals

Kubernetes’ Horizontal Pod Autoscaler can use custom or external metrics, so CPU being a poor trigger isn’t a reason to replace it. Teams do need to build and test the metrics pipeline, scaling bounds and accelerator capacity, and no policy makes GPUs appear instantly. Looking to 2027, Ramachandran points to content-aware routing, co-scheduling and disaggregated serving. An experimental Gateway API proposal cites a router that reads the model name from a request body, but benefits should be measured on your own traffic.

5. Treat Prompts as Data

Prompts, responses and retrieved documents can end up in logs and traces. “Prompts are data,” Bogoevici says, so teams should decide what to collect, where it goes, how long it’s kept and who sees it.

Where It Doesn’t Apply

Teams calling a hosted model API can’t see GPUs at all, and their visibility stops at the provider boundary. Stack Archive’s look at EKS Fargate internals is a reminder that GPUs aren’t available on every Kubernetes compute option either.

Source: The New Stack

Content created with AI assistance and reviewed for accuracy.