GitHub ReviewBench: Copilot Wins Its Own AI Code Review Test

GitHub built a benchmark for AI code reviewers, ran every competitor through it itself, and put its own product in first place. That is not necessarily wrong, but on the one independent leaderboard with live data, the same product ranks fourth. For a platform team picking a reviewer to wire into every pull request, the useful question is not who won. It is why the rankings disagree.

What Happened

GitHub unveiled ReviewBench, an open benchmark developed with Microsoft, to measure how well AI code review agents find useful problems in pull requests. It uses 219 public pull requests from 187 repositories across 19 languages. GitHub says it chose the corpus after analyzing 103.9 million pull requests, deliberately weighting it away from tiny single-file changes.

The reference set of “correct” findings combines human review comments, later author changes, static-analysis output, and LLM reviewers. Claude Sonnet 5 classifies findings and a separate LLM matcher pairs candidate findings with known issues. Human and classifier judgments agreed 96.6% of the time. Copilot code review, in its default Balanced mode, leads with a 40.1% grounded F1 score, The New Stack reports.

Why It Matters

The caveats are substantial, and GitHub states them itself. Its own team generated the initial entries by running public versions of each product. No vendor ran or verified its own result. Products were also tested on different dates: Copilot on October 1, Cubic and Greptile back in June. ReviewBench warns that tools may have changed since and that corpus performance may not carry over to a company’s own code.

Martian’s Code Review Bench, an independent effort launched in February, tells a different story. As of October 6, its online leaderboard, built from how developers actually respond to review comments in real repositories, ranks Cubic first at 64.9% F1, then Greptile and CodeRabbit, with Copilot fourth at 60.9%. On Martian’s offline leaderboard, which uses an F2 score weighted toward recall, Qodo Deep leads and Copilot sits fifth at 58%.

The numbers are not directly comparable: different datasets, different scoring, different evidence. That is the lesson. Martian found at launch that its own offline results diverged from real-world behavior, which is why it made the online metric its headline. Benchmark design is choosing the winner as much as the products are.

What This Means for Builders

Code review is now a CI/CD component with real cost. Copilot code review bills Actions minutes on private repositories and can approve pull requests. A reviewer that floods PRs with low-value comments taxes every engineer. One that misses issues while carrying approval rights is worse. Precision-weighted and recall-weighted metrics reward opposite failure modes, so pick the one that matches what the team can tolerate.

ReviewBench does have a credible path to usefulness. GitHub publishes the dataset, methodology, and judging setup, and vendors can submit their own runs. If competitors start submitting, the leaderboard becomes vendor-tested rather than GitHub-tested. For context on how Copilot’s billing model has shifted this year, see Stack Archive’s coverage of GitHub Copilot token billing.

Source: The New Stack

Content created with AI assistance and reviewed for accuracy.