AWS DevOps Agent + SageMaker HyperPod: Automated Incident Triage at ML Scale

SageMaker HyperPod handles the hardware faults that interrupt distributed training jobs: it detects faulty accelerators, replaces them automatically, and resumes the job without operator involvement. That self-healing capability works well for the scenarios it covers. The scenarios it does not cover — configuration errors, capacity constraints, pods stuck in CrashLoopBackOff, workers marked NotReady — still require human triage. Typically 30 to 45 minutes of human triage before an operator even knows whether the issue requires intervention or will resolve on its own.

How It Works

The solution pairs HyperPod’s Health Monitoring Agent with an AWS DevOps Agent configured with two skills: triage and root cause analysis. The agent operates in read-only mode — it investigates and classifies, it does not take remediation actions — and delivers verdicts by email via Amazon SES.

Two detection paths feed incidents into the agent. The first is event-driven: HyperPod emits cluster events to Amazon EventBridge, a webhook bridge Lambda filters out informational noise and retains warnings and errors, signs payloads with HMAC-SHA256, and POST them to the DevOps Agent. The second is polling-based: a Lambda audits Kubernetes state every 15 minutes for CrashLoopBackOff pods and NotReady nodes.

The triage skill decides whether to link an incoming event to an existing investigation, skip it (already handled), or open a new investigation. The RCA skill reconstructs timelines from logs and event sequences, then classifies the incident as Suppress, Monitor, Escalate, or Resolved. The four-level classification gives on-call engineers a pre-triaged verdict rather than raw alert data.

What This Means for Platform Teams

The cost profile is a meaningful architectural signal. On a healthy cluster, the solution’s daily heartbeat costs approximately $30-60 per month. Each investigation costs approximately $4 and takes about 8 minutes. For a team running multiple HyperPod clusters at scale, this shifts triage from a recurring engineering cost to a predictable per-investigation infrastructure cost — and removes the human-in-the-loop requirement for the investigation phase entirely.

The filtering architecture is what makes this viable rather than noisy. HyperPod events are numerous; one hardware failure can generate 5-10 EventBridge events. The webhook bridge and EventBridge filter layers deduplicate before anything reaches the agent, so operators see one investigation per incident rather than a flood of correlated alerts.

The design also separates detection logic (in Lambda code) from reasoning logic (in plain-English skill documents), making both independently maintainable. A new detection pattern — a new failure mode, a new event type worth monitoring — requires only a Lambda update, not a change to the agent’s reasoning.

Source: AWS DevOps Blog

Content created with AI assistance and reviewed for accuracy.