Agent Evals: A 5-Step Framework From LangChain's LangSmith Team

In LangChain’s own test, Jev, a judge model from TypeSafe AI, matched human labels on every pass/fail decision for $0.34 in total, against $28.17 for Claude Sonnet 4.6. That figure sits in LangChain’s answers to the most common questions about agent evals, and it points at the post’s larger argument: teams shipping agents don’t need a big benchmark to start testing them, just a handful of real traces and a loop. For engineers whose agents already have users, the post is a practical starting order.

The Framework

Agents are harder to test than traditional software for three reasons LangChain lists: outputs vary between runs, more than one answer can be correct, and a correct output can come from a wrong path, such as calling a tool the agent didn’t need. The framework below follows the post’s recommended sequence, using its travel-assistant example.

Start With One Behavior and 10 to 20 Traces

Read a sample of traces first, then pick one behavior to test, such as whether a travel assistant books the requested date and price range. Ten to twenty examples is enough to start. Run the current agent against them and save the result as a baseline experiment; every later version is compared with it. Each new failure from testing or production becomes another example.

Run Offline Evals Before Shipping, Online Evals After

Offline evals run the agent against a fixed dataset before a change ships, which catches regressions from a new prompt, model or tool. Online evals score real production traces as they arrive, usually with no reference output. LangChain’s point is that they feed each other: failures flagged online become offline examples.

Pick the Cheapest Evaluator That Works

The post names four evaluator types: code checks (fast, cheap, consistent), LLM-as-a-judge for what code can’t check, human review as the best signal for subjective quality, and pairwise comparison for choosing between two prompts or models. To trust a judge, compare it with human scores on a sample, ask narrow questions (“Does the reply mention every layover?”), and use binary or labeled scales rather than 1 to 10.

Score the Path and the End State, Not Just the Answer

Multi-step agents can produce a fine reply while the conversation goes wrong. LangChain recommends grouping turns into threads and scoring the whole conversation, and for agents that change things, checking the final state. For a coding agent asked to fix a bug, the reliable check is running the repo’s tests in a clean copy after the agent finishes. Harbor, the open-source harness behind Terminal-Bench, builds a container per task for this.

Control the Cost of Online Evals

Online eval costs grow with traffic if a judge scores every trace. The post suggests sampling a percentage of traffic or filtering to traces with negative feedback, using code checks where possible, and using a cheaper judge, the source of the $0.34 result above.

Where It Doesn’t Apply

This is a vendor guide, and every tool it recommends, from annotation queues to automatic eval writing, runs in LangSmith. The concepts are portable, but the $0.34 judge result comes from LangChain’s own experiment, and the post cites no independent reproduction. Teams without production traffic also can’t use online evals or trace-mined examples; LangChain’s fallback is dogfooding with internal testers or trusted design partners. For finding failure clusters across traces, Stack Archive covered LangSmith AI Search: Finding Agent Failures by Behavior.

Source: LangChain Blog

Content created with AI assistance and reviewed for accuracy.