CircleCI's Framework for Testing AI Agents in CI: Contract Tests and Agent Evals

As AI agents move from demos to production systems, teams are discovering a validation gap: they have CI pipelines for code, but nothing equivalent for agent behavior. An agent’s experience (AX) — whether it can successfully complete its core workflows, how efficiently it uses tools, how often it recovers from errors — isn’t covered by unit tests or standard integration checks.

CircleCI’s framework addresses this with a two-layer approach that fits into existing CI infrastructure without requiring specialized observability tooling.

The Two Layers

Layer 1: Deterministic Contract Tests

Contract tests validate the interfaces that agents depend on. These are binary pass/fail checks that should run as hard gates in CI — if they fail, no merge proceeds.

What belongs here:

• API schemas — does the endpoint return the expected structure?
• CLI output format — does parsing --json output still work after a dependency bump?
• MCP tool definitions — do the tool descriptions and parameter schemas match what the agent was trained to expect?
• Error codes — does a 404 still mean “not found” or did something upstream change its behavior?

These tests protect the surface area between your product and the agents using it. CircleCI describes the guiding principle as “fix the interface, not the prompt” — if agents start failing because your API changed a field name, the right fix is making the API contract stable, not adjusting agent instructions to handle the inconsistency.

Layer 2: Agent Evals

Agent evals measure workflow quality. Unlike contract tests, these aren’t binary — they use baseline thresholds and track metrics over time.

Metrics worth capturing per workflow:

• Completion rate — does the agent successfully finish the task in representative test cases?
• Tool selection efficiency — is it calling the right tools in the right order, or taking unnecessary detours?
• Retry count — how often does the agent fail and retry before succeeding?
• Latency and token consumption — is efficiency degrading as the product evolves?

The distinction from contract tests: evals tell you whether the agent’s behavior has regressed, not whether the interface it depends on is intact. Both can degrade independently. A passing contract test doesn’t guarantee good agent behavior; a passing eval doesn’t mean the underlying interface is stable.

Running Cadences

CircleCI recommends running different validation types at different cadences:

The rationale for not running full evals on every PR: they’re computationally expensive and may have signal-to-noise issues if the baseline thresholds aren’t well-calibrated yet. Starting with high-impact workflows where failures have real costs — and adding coverage as you learn what degrades — is more practical than attempting comprehensive eval coverage from day one.

Netlify’s AXIS framework is referenced as a real-world example of this structure in production.

The “Fix the Interface” Principle

The most practically useful guidance in CircleCI’s framework is the emphasis on interface stability over prompt compensation. When an agent starts underperforming, the instinct is often to update the system prompt — add more context, clarify instructions, try a different framing. Sometimes that’s the right fix. More often, something in the environment changed: an API response format shifted, a tool description drifted out of sync with actual behavior, an error message changed.

Contract tests surface those environmental regressions before they reach production. They also create accountability: if merging a backend change breaks agent behavior, the contract test failure makes the connection explicit rather than leaving it as an unexplained eval decline a few days later.

Source: CircleCI Blog

Content created with AI assistance and reviewed for accuracy.