Amazon ECS GPU Auto Repair: How It Replaces Failing GPUs

A failing GPU rarely takes a host down cleanly. It stops processing, drops off the bus, or starts throwing memory errors while the instance still looks healthy to the scheduler. Amazon ECS now treats those failures as a reason to replace the machine, and it does so by default.

How It Works

GPU auto repair applies to Amazon ECS Managed Instances with NVIDIA GPU hardware. ECS uses NVIDIA Data Center GPU Manager (DCGM) to monitor GPU health. When DCGM reports a critical failure, ECS marks the instance as impaired.

The trigger list is specific. AWS documents 18 NVIDIA Xid error codes that cause an instance to be marked impaired and replaced, including Xid 48 (double bit ECC error), Xid 74 (NVLink error), Xid 79 (GPU has fallen off the bus), and Xid 140 (unrecoverable ECC error). NVIDIA’s Xid documentation describes what each code means.

Replacement follows a start-before-stop sequence:

• ECS sets the impaired instance to DRAINING, so no new tasks land on it.
• ECS provisions a replacement instance.
• Existing tasks stop gracefully, and ECS honors each task’s stop timeout.
• After the drain period ends, ECS terminates the impaired instance.

Two guardrails keep repairs from turning into an outage. No more than 20% of a capacity provider’s instances can be draining at once, and capacity providers with fewer than 9 instances drain at most one instance at a time. Health is visible through the DescribeContainerInstances API and through container instance health change events in EventBridge.

AWS first announced the feature in April 2026, and a September 2026 update extended it to the China (Beijing) and China (Ningxia) Regions at no additional cost. The New Stack revisited it this week from the SRE angle.

What This Means for You

The main change is who owns the first response. Without auto repair, a GPU that falls off the bus becomes a page, a manual drain, and a replacement. With it, the platform handles the common hardware failures and the on-call engineer reviews an event afterward.

There are trade-offs to plan for:

• Opting out has side effects. Setting actionsStatus to DISABLED in autoRepairConfiguration stops automatic replacement while keeping monitoring on. It also disables Amazon ECS Managed Daemons auto repair.
• Small fleets heal slowly. With fewer than 9 instances, ECS replaces one host at a time. Two bad GPUs in a five-node training pool means serial repair.
• Stop timeouts matter more. Graceful drain only helps if long-running inference or training tasks checkpoint inside their stop timeout.
• Capacity has to exist. Start-before-stop needs a replacement instance, so scarce GPU types and tight quotas still decide how fast a repair finishes.

Teams running GPU workloads on Kubernetes instead face the same failure modes with more assembly required. Stack Archive’s guide to building an AI factory on Kubernetes covers that side of the comparison.

Source: AWS Documentation

Content created with AI assistance and reviewed for accuracy.