
Karpenter 1.15: GPU-Aware Provisioning and Repair That Reboots
Cloud Platform EngineeringUntil now, a Karpenter repair could drain pods faster than a Pod Disruption Budget allowed, and a GPU node with a transient fault was replaced outright, sometimes losing its capacity reservation slot. Karpenter 1.15 changes both. For platform teams running distributed training or stateful services on EKS, it’s the release where node repair finally plays by the same availability rules as everything else.
What Happened
AWS’s v1.15 walkthrough groups the changes into three areas: provisioning that understands hardware, repair that respects workloads, and consolidation that lines up with your own scaling. Several of them ship behind feature gates, so upgrading alone doesn’t switch them on.
The headline provisioning change is Dynamic Resource Allocation (DRA) support in the AWS provider. GPU and Elastic Fabric Adapter (EFA) workloads need both devices on the same PCIe root complex so data moves peer to peer without going through the CPU and system memory, which matters for collective operations like AllReduce. Generic resource counts can’t express that. With DRA, Karpenter evaluates device availability and topology before it requests a node. It requires EKS 1.34 or later, GPU time-slicing needs EKS 1.36, and it’s enabled with the DRANVIDIAGPU and DRAEFA gates or settings.enableDRA in Helm. The Karpenter DRA docs cover the setup.
Why It Matters
Repair is now a voluntary disruption. That means it honors NodePool disruption budgets, respects PDBs during drains and applies a per-condition termination grace period. AWS says earlier releases could exceed the intended concurrency budget or drain pods faster than a PDB permitted.
Repair can also reboot instead of replace. Policies can match on a condition reason and choose RebootNode or ReplaceNode. On AWS, the EKS Node Monitoring Agent flags some GPU XID errors as reboot-clearable, which triggers an in-place reboot, keeps the node in the fleet and preserves the capacity reservation slot. That path is alpha, behind the RebootForRepair gate, and the controller role needs ec2:RebootInstances. For fleets at their node limit or on a full Capacity Reservation, alpha TerminateFirstDrift and TerminateFirstRepair gates remove the old node before launching its replacement, since there’s no room to stage one first.
What This Means for Builders
Two smaller fixes remove long-standing annoyances. Karpenter now checks EBS volume attach limits before requesting an instance; AWS gives the example of an m5.xlarge supporting up to 25 volumes. Previously, PVC-heavy workloads could trigger a loop of launching nodes that couldn’t attach enough volumes. Nitro Enclaves are supported through a new spec.enclaveOptions field on EC2NodeClass, so confidential computing workloads no longer need a static node group outside Karpenter.
On consolidation, a PodDeletionCostManagement gate makes Karpenter manage the pod-deletion-cost annotation so Deployment scale-down removes pods from the nodes it’s trying to empty. A NodeClassCEL gate lets one EC2NodeClass compute maxPods, kubeReserved and systemReserved per instance type with CEL expressions, instead of maintaining parallel NodeClasses.
Before upgrading, AWS points to the upgrade guide for breaking changes, including lowercase values for the reason metric label and a changed consolidation_type label on queue_failures_total. Dashboards and alerts keyed on those labels will need updating. Teams already running GPU repair on ECS can compare it with ECS’s GPU auto repair.
The So What
EKS teams running training jobs or strict-PDB services should upgrade to the latest 1.15 patch, fix any metric-label queries first, and turn on repair-as-voluntary-disruption before anything else. Test RebootForRepair on one GPU NodePool with reserved capacity, since that’s where keeping the reservation slot pays off most, and leave the alpha terminate-first gates off unless your fleet is at its node limit.
Source: AWS Containers Blog
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →