GitHub Secret Scanning: Why a Small Model Beat the LLM Pipeline

GitHub replaced the large language model in its generic secret detection with a much smaller classifier, and serving costs fell by more than 98%. GitHub’s engineering write-up says the switch also cut median post-push scan time by 86% and raised owner-weighted alert acceptance by 11.5%. For platform and security teams who have learned to ignore secret scanning alerts, the bigger change is where the faster model now runs: before code is pushed, not only after.

How It Works

Structured credentials often have a distinctive prefix, length, or checksum, so a regex can find them reliably. Generic secrets don’t. A password-shaped string might be a production credential beside a database URL, or the same value in a test fixture. A high-entropy string might be an API key, or a checksum, build ID, or asset hash. GitHub calls these lookalikes hard negatives, and context is the only way to tell them apart.

The new scanner splits the job in two:

• Candidate generation. Regex patterns scan content with high recall, flagging anything that looks like a credential so real secrets aren’t missed at the start.
• Context-aware classification. A fine-tuned ModernBERT model reads each candidate with its file path, secret type, variable names, and a bounded window of surrounding code, then scores how likely it is to be real. High-confidence findings become alerts; placeholders, fixtures, and checksums are filtered out.

GitHub’s point is that the model doesn’t replace deterministic scanning. Regex stays the fast, transparent way to find candidates, and the classifier is used only where judgment is needed. Training data combined product feedback with human validation, high-recall regex candidates with human and weak labels, and synthetic examples aimed at hard cases such as documentation secrets and near-miss pairs. GitHub grouped rows sharing a repository or candidate value before splitting the data, so near-duplicates couldn’t leak between training and evaluation. The model is exported to ONNX with 8-bit quantization.

Offline, on the same public traffic, the new pipeline had 13% higher precision and 19% higher recall than the previous LLM pipeline. A customer A/B test then produced the 11.5% acceptance gain. GitHub weights acceptance by repository owner so customers who bulk-resolve large volumes of alerts don’t distort the average.

What This Means for You

Speed is what moved generic detection into push protection. Because the classifier is fast, GitHub can now return generic-secret findings before a push is accepted, giving developers a chance to fix a leak before the secret enters the repository. GitHub’s companion post on secret protection frames this as keeping pace with AI-accelerated development, and the secret scanning documentation covers how push protection is configured.

The design lesson travels beyond GitHub. In GitHub’s words, “model size is not the same as model usefulness”: a narrow, measurable decision can be served better by a small fine-tuned encoder than by a general-purpose LLM. Teams running open-source scanners such as Betterleaks face the same precision problem, and the regex-plus-classifier split is a pattern they can copy.

Source: The GitHub Blog

Content created with AI assistance and reviewed for accuracy.