The Definitive Guide to Building an AI Factory on Kubernetes for Platform Teams

The core tension in multi-tenant GPU infrastructure is not technical — it’s economic. GPUs are the dominant capital expense in any AI platform. The only way to justify that spend is high utilization across many workloads. But high utilization across many teams means shared accelerators, and shared accelerators create a hard isolation problem: how do you run simultaneous fine-tuning jobs, inference endpoints, and model evaluations on the same physical hardware without one team’s workload bleeding into another’s?

That is the question an AI factory on Kubernetes is designed to answer.

The Core Problem

GPU utilization waste starts at the allocation layer. Traditional Kubernetes device plugins are binary: a pod requesting one GPU occupies the entire accelerator, regardless of actual compute consumption. A fine-tuning job using 10% of a GPU’s capacity still locks out every other workload on that card. At typical enterprise AI workload densities, this means the majority of purchased GPU capacity sits idle while pods queue for scheduling.

Platform teams are caught in a bind. They can provision more hardware to reduce queue depth — expensive. Or they can attempt multi-tenancy — complex, risky, and difficult to audit. The AI factory model is a structured answer to that second path: one fleet, many teams, different quotas, policies, and trust boundaries, all operating on pooled GPU resources. The CNCF community has been documenting this architecture in detail at cncf.io, and the component choices are maturing quickly.

The Nine-Layer Architecture

The full AI factory stack runs nine layers from hardware lifecycle management through observability and security. Platform teams don’t build all of them — but they own the integration surface between them.

The layers that carry the most operational weight:

Node Inventory — Node Feature Discovery (NFD) and the NVIDIA GPU Operator handle hardware detection, driver lifecycle, and DCGM exporter deployment. This is largely commoditized; the GPU Operator handles what used to require manual node bootstrapping.

Tenant Isolation — vCluster gives each team a virtual Kubernetes control plane: its own API server, RBAC policies, and admission webhooks. From the tenant’s perspective, they have a dedicated cluster. From the platform team’s perspective, it’s a namespace-scoped process on shared infrastructure.

GPU Allocation — Three mechanisms, matched to trust level: whole-GPU allocation for untrusted tenants (dedicated physical accelerator, no sharing), fractional allocation via HAMi (a CNCF Incubating project that enforces per-pod memory and compute limits in software) for trusted internal teams, and NVIDIA MIG for workloads requiring hardware-level memory and fault isolation. The allocation mechanism is a policy decision, not a one-size configuration.

Workload Execution — vLLM and KServe handle inference serving. KAI Scheduler and Volcano manage topology-aware gang scheduling for distributed training runs. Kueue sits at the admission layer, enforcing quota and preventing any single tenant from flooding the scheduler.

Tenant Isolation in Practice

The vCluster control plane handles namespace-scoped resource governance, but true isolation requires enforcement at both planes.

On the data plane: network segmentation via VPCs, VXLAN/EVPN overlays, and per-tenant storage quotas enforced through Pod Security policies. On the control plane: per-tenant admission webhooks that reject workloads violating resource policy before they reach the scheduler. The result is that a misconfigured pod in one tenant’s virtual cluster cannot affect scheduling pressure, network bandwidth, or storage capacity for any other tenant.

The hardest production challenge, per teams who have shipped this architecture, is balancing density, isolation, and chargeback accuracy without hiding the GPU data path behind an abstraction. Accurate FinOps on shared GPU infrastructure requires visibility down to the fractional-GPU level — which is exactly what HAMi’s per-pod memory and compute tracking is designed to provide.

The Build vs. Assemble Decision

Platform teams evaluating this architecture face a fork: NVIDIA DSX OS (an integrated stack that handles most of these layers) versus composing from CNCF components directly.

NVIDIA DSX OS reduces initial integration complexity and provides a single vendor support path. The cost is potential lock-in and reduced flexibility when your requirements diverge from NVIDIA’s design choices. The CNCF component path — NFD, GPU Operator, HAMi, vCluster, Kueue, Volcano — is more operationally complex but gives teams precise control over each layer and avoids coupling the entire platform to one vendor’s release cadence.

For teams already running mature Kubernetes platform operations, the CNCF path is usually the right call. For teams new to GPU infrastructure who need to move fast, NVIDIA DSX OS may be worth the lock-in risk to compress time-to-production.

Content created with AI assistance and reviewed for accuracy.