Grafana Watchers: The Design That Keeps Always-On AI Agents Cheap

An AI agent that investigates your environment every 10 minutes runs 4,320 times a month. At the $0.56 to $1.26 Grafana Labs measured per agentic investigation, that is “thousands of dollars a month” for one monitor. Grafana’s new Watchers do the same always-on job for less than $5 a month at the median, and the engineering behind that gap is a template for any platform team putting agents on a schedule.

What Happened

Grafana made Watchers generally available at ObservabilityCON 2026 as “always-on agents in Grafana Assistant.” A user describes what to monitor in natural language and picks data sources, and the watcher learns what normal looks like. Data sources include Prometheus, CloudWatch metrics, Loki, Tempo, Pyroscope, BigQuery and any JSON or XML API through the Infinity data source. Findings go to Slack, Microsoft Teams, signed webhooks or Grafana Alerting. Per the Watchers documentation, they live under “AI > Watchers” in Grafana Cloud.

How It Works

William Dumont’s engineering post splits the job into three layers, and only one of them is expensive.

Calibration runs once, and again on recalibration. A background agent with a capable model maps the telemetry, asks the user scoping questions, mines existing alert rules and dashboards, and measures each signal over the past week to capture daily and weekly cycles. Every check is tested against live data. The output is a compact “calibration context,” and the post says calibration takes “a few minutes at most.”

Per-run data work is plain code. Metrics are compared with the same time yesterday and with a statistical baseline. Logs go through Loki’s patterns API, which tracks new patterns and patterns whose rate doubles. Deploy checks tell the model whether a change started right after a rollout.

The per-run decision is a single call to a small, fast model with no tools. It returns OK, warning or critical, a short report, tracked issues with evidence, and whether to notify or launch a full investigation. Grafana says that model “didn’t miss a single critical case, at about half the cost per decision” of the one it replaced, based on evals that replay production decisions.

What This Means for You

The results are specific. Of Grafana Cloud’s 43 internal watchers, 41 cost under $10 a month. The most expensive, with 43 checks every 15 minutes, costs “a little over $20,” and none spends more than a cent per run, at a $2 per million token list price. The team’s own watcher found more than 10 customer-affecting production bugs over a few months. The post also admits a miss: during one rollout it flagged MySQL exhaustion but blamed the wrong cause.

Watchers are free until December 1, 2026, then draw on a plan’s system-initiated allowance and bill on Assistant token usage under Assistant pricing. Stack Archive’s coverage of ObservabilityCON 2026 covers the wider release.

Source: Grafana Labs

Content created with AI assistance and reviewed for accuracy.