METR's New Metric Shows Exactly When an AI Agent Costs More Than a Human

Every AI-versus-human productivity claim runs into the same missing number: at what budget does the AI stop being the cheaper option? METR built a metric to answer that directly, and when it tested current frontier models against real human engineering effort, most of them never got there.

How It Works

METR calls the metric the “expenditure horizon” — the spending threshold at which an AI agent and a human worker cost the same to produce an equivalent improvement. Below that threshold, the agent is the economical choice; above it, the human is. The metric doesn’t just count API bills. It totals three cost categories: the AI’s own operating spend, the compute burned running experiments, and the human labor time the comparison is measured against.

METR tested this on the NanoGPT speedrun project, an open effort to optimize GPT-style model training as fast as possible. To establish the human side of the ledger, METR’s researchers interviewed contributors and used AI models to estimate the actual effort behind each improvement — arriving at a baseline of roughly $2,500 per one-percent speedup, based on an estimated $150 hourly rate and about 16 hours of skilled work per gain. Total human effort invested in the NanoGPT project across its history comes out to an estimated $250,000.

Against that baseline, METR ran a slate of models — GPT-5, GPT-5.2, GPT-5.5, GPT-5.6 Sol, Opus-4.1, Opus-4.8, Opus 5, and Fable 5 — and measured their expenditure horizons directly. Most produced horizons close to $0, meaning they couldn’t reliably produce a real improvement at any budget. Only two cleared a meaningful bar: GPT-5.5 reached a 1% improvement, and Opus-4.8 reached 1.5%, with expenditure horizons in the $0–$3,300 range.

What This Means for You

This is a more honest benchmark than the productivity claims usually circulating — including headline figures like AI agents completing 16% of freelance work at professional quality — because it forces a like-for-like cost comparison instead of a completion-rate anecdote. “AI wrote code 10x faster” means little if the code required 10x the review time to trust. An expenditure horizon converts that into a single number: the price point past which you should stop asking the agent and start asking the engineer. For teams deciding where to point autonomous agents versus where to keep a human in the loop, that’s a more actionable signal than a leaderboard score, because it’s denominated in the currency that actually constrains a budget.

The result is also a check against the industry’s implicit assumption that model releases are steadily converting more of the economy into “AI does it cheaper” territory. METR’s data shows that for a hard optimization problem like speeding up model training, that conversion has barely started — the horizon is still measured in low thousands of dollars on a project that consumed a quarter-million dollars of human effort.

Source: The Decoder

Content created with AI assistance and reviewed for accuracy.