Amazon CloudWatch Omni: Collaborative AI Observability Built for Agentic Systems

Traditional CloudWatch assumes you’re monitoring infrastructure signals — alarms, dashboards, metrics organized by service. CloudWatch Omni assumes your stack now includes AI agents with their own reasoning traces, tool invocations, and evaluation metrics. The two assumptions require different products, and AWS has built them separately.

What Happened

CloudWatch Omni launches as a standalone web application separate from the AWS Management Console, with identity provider integration (IAM Identity Center, Okta, Azure AD) that lets the full incident response team access a shared investigation URL without individual Console credentials. Two distinct capability layers shipped simultaneously:

For general applications: Automatic service and dependency topology discovery from telemetry data — the application map updates dynamically as services deploy. Amazon DevOps Agent participates in investigation sessions, correlating signals across services and building root cause paths. Incident escalation preserves full context as team members join; investigation history captures automatically for post-incident review.

For generative AI and agentic workloads: A dedicated trace exploration layer with Trace Explorer (hierarchical timelines of LLM calls, tool invocations, and reasoning steps), Compare Mode (side-by-side trace comparison across prompt or config changes), Session Explorer (full multi-turn conversation history), and Agent Topology (system architecture including sub-agents, tools, and interconnections). The evaluation framework includes 17 built-in evaluators covering coherence, helpfulness, faithfulness, and routing correctness, with third-party integrations for AutoEval and DeepEval.

Framework support spans LangChain, LangGraph, CrewAI, OpenAI SDK, Strands, and Vercel AI SDK in Python and TypeScript. Native integration connects to Amazon Bedrock AgentCore and its evaluation capabilities.

Why It Matters

The architectural decision to build Omni as a separate application — with its own URL, its own identity model, and application-centric rather than signal-centric organization — addresses a structural problem with traditional CloudWatch for team-based incident response. When an incident requires an SRE, a developer, a database engineer, and a manager, the CloudWatch Console’s per-resource signal organization creates friction. Omni’s shared investigation sessions with DevOps Agent participation represent a meaningful change to how incident resolution actually works in practice.

For teams running AI agents in production, the 17-evaluator framework and Trace Explorer are the concrete tools that make “observing an LLM in production” more specific than “check the logs.” Routing correctness as a named metric is particularly relevant for multi-agent systems where agent selection failures are common but hard to debug with generic observability tools.

What This Means for Builders

Two immediate assessments for teams currently using standard CloudWatch. First, if you’re running agentic workloads on AWS — Bedrock, Lambda-based agents, ECS agent services — the Trace Explorer and evaluation framework represent the closest thing to a native AWS solution for LLM observability. Purpose-built tools like LangSmith and Datadog exist in this space, but the native Bedrock integration path in Omni is simpler for AWS-native stacks. Second, the DevOps Agent participation in investigation sessions requires the agent to have context about your application topology — the automatic discovery from telemetry is the prerequisite, and getting that instrumentation right is the actual work.

Source: AWS News Blog

Content created with AI assistance and reviewed for accuracy.