GitHub Rubicon: Git Storage Built for 5,000 Concurrent Agents

The busiest repositories on GitHub take at most dozens of writes a second today, in short bursts. GitHub expects agents working in parallel around the clock to turn those bursts into a sustained rate within a year, and one repository’s traffic already grew about 4x between March and May. So GitHub set a target of two orders of magnitude more scale per repository, and in internal testing its new Git storage architecture handled 5,000 concurrent agents sending 7,200 pushes a second to one repo, with consistently low read latency.

How It Works

GitHub’s engineering post describes the design, code-named Project Rubicon, as the third step in a series. The previous post covered moving nearly every repository into Azure and scoping the write quorum to individual references, which Stack Archive covered here.

The key observation is that very little of a write needs ordering. Git objects are immutable and named by their contents, so two writers can store them at once without agreeing on anything. Only the reference update needs agreement, and it is tiny. Rubicon builds on that:

• Spokes still routes requests, balances load and keeps tenant partitioning and fairness.
• Virtual replicas are disposable compute serving reads, writes or both. Their local disk is a rebuildable cache, and busy repositories get more of them.
• Azure Blob Storage holds packs, indexes, read accelerators and cached fetch responses.
• Ref-CAS, a replicated log service, holds references and the pack manifest.
• Compaction nodes merge packs and remove unreachable objects in the background, never on a virtual replica.

A write indexes the incoming pack, uploads it to Blob Storage, then commits one Ref-CAS transaction that lists the pack and updates the references. A read checks its reference version with Ref-CAS and pulls only the packs it lacks.

GitHub first tried keeping references in Blob Storage. A single write-ahead log blob capped writes in the mid-hundreds per second, because only one conditional write to a blob can be in flight. Splitting references across segment blobs raised the write ceiling but made every read check every segment, trading write latency for read latency when reads outnumber writes by orders of magnitude. Ref-CAS instead commits each reference update as a compare-and-swap by quorum, locking only the references it touches. Agents mostly push to their own branches, so most updates don’t contend. An unchanged repository costs a single version comparison, and reference transactions run in the tens of thousands per second.

Compaction runs per shard under a lease, so adding shards and nodes adds throughput. The result, GitHub says, is an effectively flat pack count and a flat latency curve as writes climb.

What This Means for You

For developers, Git stays Git: push, fetch and branch work the same, with the same protections. The architecture is in limited private preview now, with a broader rollout early next year.

The roadmap is where agent builders should look. GitHub lists working without a clone (reading a few files straight from the server), an atomic multi-file commit API that lands or rejects a change cleanly, and event streams agents can subscribe to. In Git itself it wants lightweight clones that work together, faster history walks via metadata like the commit-graph, and warnings for common mistakes before they cause damage.

Source: The GitHub Blog

Content created with AI assistance and reviewed for accuracy.