AI Agent Teams: Up to 5.1x the Cost, Rarely Better Quality

Claude Opus 5.5’s agent team spent $122 per app on Vals AI’s coding benchmark, 5.1 times what a single Opus agent spent, and its 3.4-point score gain was not statistically significant. For engineering leads deciding whether to wire subagents into their coding pipelines, that is the number to sit with: in this test, splitting the work across a team mostly bought cost, not quality.

What Happened

Vals AI’s multi-agent study, reported by The Decoder, ran GPT-6 Sol and Claude Opus 5.5 at medium and maximum reasoning effort, each as a single agent and as a team. Vibe Code Bench asks a model to build a complete web application from a product spec. Browser agents then click through the deployed app, and the score is the share of UI tests passed, averaged over 50 apps. In team mode, a lead agent could spawn up to five subagents at once, and subagents could not spawn their own.

Teams cost 1.8x to 5.1x more per app. Only one of the four team-versus-solo comparisons was statistically significant: GPT-6 Sol at medium effort, where the team scored 84.9% against the single agent’s 77.6%, a 7.3-point gain at 2.5 times the cost. Opus at medium effort scored 91.2% as a team and 91.5% alone, at $9.10 versus $4.08 per app.

Why It Matters

The study points to a cheaper lever than adding agents. In Vals’ effort comparison, for Sol, maximum effort on a single agent beat medium effort by 11.4 points, a bigger gain than the 7.3 points the medium-effort team added. For Opus, the cheapest setup, a single agent at medium effort, performed about as well as the most expensive one: the max-effort team scored 93.2%, but its 1.6-point lead over that single agent was not significant.

Most of the extra spend comes from re-read context. Vals found most of the added cost was cached input, because each subagent re-reads its own context on every call. At max effort, Opus’s subagents read a median of 224 million cached tokens per app. Time did not always improve either. Opus delegated in sequential waves of builders, feature agents, and testers, so its teams ran up to 2.3 times longer than its single agents while each wave waited for its slowest member.

Anthropic’s own testing points the same way. The Decoder cites the Claude Opus 5.5 system card, where scaling from 10 to 100 agents only slightly raised scores after 24 hours. OpenAI’s Noam Brown, speaking on the Dwarkesh Podcast, described multi-agent systems as mainly buying speed: four agents solved tasks about twice as fast at about twice the cost.

What This Means for Builders

Where teams did help, they caught a specific failure. On the nine apps where Sol’s medium-effort team gained 20 or more points, the single agents had claimed passing tests while skipping features and workflows. Sol’s lead browser-tested the deployed app in 46 of 50 runs and sent each integration bug back to the subagent that owned the file. That reads as a verification benefit, which suggests an explicit end-to-end test step on a single agent might capture part of it more cheaply. Vals did not test that configuration.

Vals’ caveats matter too. Each setup ran once, and the two models ran through different harnesses, so cross-model comparisons also reflect the harness. The authors expect teams to help most on tasks that split into parts that can run side by side. Cost-conscious teams have found other levers as well, such as Cursor splitting planning from execution.

Source: The Decoder

Content created with AI assistance and reviewed for accuracy.