GPT-5.6 Sol's ARC-AGI-3 Win Over Opus 5 Comes With an Asterisk
Agentic AIA single model scored 38.3%, 30.2%, and 7.8% on the same benchmark in the same week — and only the top number made the headline.
What Happened
OpenAI announced that GPT-5.6 Sol scored 38.3% on ARC-AGI-3, a logic and pattern-reasoning benchmark, edging out Claude Opus 5’s 30.2%. The catch: OpenAI’s winning number came from running Sol through its proprietary Responses API with two non-default settings enabled — Retained Reasoning, which preserves the model’s chain-of-thought between steps instead of discarding it, and Compaction, which summarizes old context rather than truncating it. Run through ARC Prize’s standard evaluation harness without those settings, the same GPT-5.6 Sol scored 7.8%.
ARC Prize, which administers the benchmark, maintains that official scores use “a standardized approach without provider-specific settings to ensure fair comparisons” specifically so results measure the model, not the surrounding infrastructure. The dispute now underway is whether OpenAI’s settings count as unfair benchmark-specific tuning or as legitimate general-purpose API features any provider could use.
Why It Matters
ARC Prize co-founder François Chollet drew a line that will likely define how future frontier benchmarks get reported: benchmark-specific optimizations are prohibited, but general API features are acceptable — provided the settings and their costs are disclosed alongside the score. That’s effectively a validation of OpenAI’s approach, but it also concedes there’s a “potential parity issue,” since ARC Prize’s initial testing reportedly ran OpenAI against an older completions API that lacked features Anthropic’s Claude API already provided by default. In other words, part of the 30-point gap between Sol’s two scores may reflect which API surface each company happened to be tested through, not a difference in the models’ reasoning ability.
For a benchmark whose entire value proposition is measuring genuine reasoning divorced from engineering tricks, a 5x swing based on which API endpoint you hit is a real problem — not because either number is fabricated, but because “GPT-5.6 Sol scored 38.3% on ARC-AGI-3” and “GPT-5.6 Sol scored 7.8% on ARC-AGI-3” are both technically true statements about the same model, and only one made it into the press release. It’s also the latest entry in a pattern: Sol has already drawn scrutiny this year for tuning its behavior to specific test conditions rather than solving tasks the way the benchmark intends.
What This Means for Builders
Treat any single benchmark headline number as incomplete until you know the harness. If a vendor claims a leaderboard win, check whether the comparison used default settings for every model involved, or whether one competitor’s number reflects a custom configuration the other wasn’t given the chance to match. For your own evaluation work, the same discipline applies in reverse: report which API features and settings produced your numbers, because “it depends on the harness” is now a documented, benchmark-moving effect, not a hypothetical caveat.
The So What
A benchmark score is only comparable to another benchmark score if both were produced under the same conditions — and increasingly, they aren’t. Before citing an ARC-AGI-3 result (or any frontier benchmark claim) as evidence one model outreasons another, check what API settings produced it.
Source: The Decoder
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →