AI Support Resolution Rates: The Best Get 70%. The Median Is 48%. Here's the Gap.

The gap between the best AI customer support deployments and the median isn’t the language model. It’s whether anyone configured the guardrails, identity checks, and escalation rules that turn a capable model into a reliable support agent.

The Number

Gorgias, the AI customer experience platform, published a benchmark of 13 AI support vendors across 212 live, mid-market ecommerce stores using real inventory — no synthetic test environments. All agents received identical customer messages. Responses were scored against 26 binary checks, with factual claims verified against live store data.

The 13 vendors tested: Rep AI, Decagon, Yuma, Ada, Gorgias, DigitalGenius, Sierra, Siena, Kodif, Intercom, Zendesk, Envive, and Klaviyo.

Rep AI led the field at 75% automation rate. The median vendor achieved 48%. The weighted average across all vendors by conversation volume was 46%. Notably, nine of the ten vendors who could show longitudinal data had lost automation rate in the final four weeks — performance is not static after deployment.

How the Math Works

The variance between 70% and 48% doesn’t trace back to model capability differences. According to the benchmark analysis, “guardrails, identity checks, and escalation rules account for much of the gap between a 45% deployment and a 70% one.”

The specific failure patterns matter:

• Policy questions scored 77.4/100 across vendors — relatively consistent, since policy questions have clear factual answers in the store’s documentation
• Order tracking scored only 60.1/100 — worse, because order tracking requires identity verification that many deployments handle poorly
• The most common failures were authentication walls (requiring verification the agent couldn’t provide), repetitive request loops, and incorrect escalation triggers — not factual errors about products

This maps directly to a configuration quality problem. A model that knows your return policy can still fail on order status inquiries if the authentication design doesn’t let it verify customer identity before accessing order data. The 70% deployments solved that. The 48% median deployments didn’t.

What It Signals

The benchmark reframes the AI customer support question from “which vendor should we use?” to “how well have we configured whatever we’re using?” The vendor results at 48% median and 70% top performance suggest there’s more variation within each vendor’s customer base than between vendors — a deployment with well-designed guardrails and escalation paths on a median vendor will likely outperform a poorly configured deployment of a top-tier model.

The longitudinal signal — nine of ten vendors losing automation rate in the last four weeks — points to a maintenance problem. AI support agents that aren’t actively updated as product inventory, policies, and customer questions evolve will degrade. The “set it and forget it” deployment is the path to the bottom of the 48% cohort, not a stable equilibrium.

For businesses currently evaluating AI support tools, the benchmark provides a useful calibration point: AI agents handling structured operational tasks perform at a fundamentally different ceiling than agents handling unstructured customer queries with identity verification requirements. Know which type of support volume you’re addressing before choosing a platform.

Source: SaaStr

Content created with AI assistance and reviewed for accuracy.