Researchers Can Now Reconstruct LLM Prompts From Outputs With Near-Perfect Accuracy

A team from IIT Bombay and Adobe Research has demonstrated near-perfect accuracy reconstructing LLM prompts directly from model outputs — without access to model weights, training data, or any internal system information. For enterprises treating system prompts as proprietary IP, this is a supply chain vulnerability hiding in plain sight.

What Happened

The technique, called Previous-Token Prediction (PTP), trains an inverse language model that learns to predict prior tokens instead of subsequent ones. The approach works by training on synthetically generated data — no access to target model weights required. Tested against GPT-4o outputs, the method reconstructed short prompts with accuracy sufficient to reproduce the original model’s responses when the extracted prompt was fed back.

The inverse model used in testing was a small Qwen-3-0.6B chatbot — a deliberately lightweight configuration, notable because it demonstrates the attack isn’t compute-intensive. The research limitation is real: testing covered short prompts of one to two sentences. Whether the method generalizes to multi-paragraph system prompts with complex instructions remains an open question.

The published paper uses the domain name stolen-thoughts.com as a demonstration vehicle, which communicates the researchers’ read on the stakes.

Why It Matters

The business model for many enterprise AI deployments rests on proprietary system prompts — the instructions, context, and constraints that differentiate one AI product from a generic API call. Companies building on top of foundation models from Anthropic, OpenAI, and Google invest engineering time in prompt engineering, few-shot examples, and safety instructions that are technically exposed every time the model generates output.

PTP doesn’t require the attacker to know which model generated the output. It works at the output layer, meaning any public-facing AI system that produces text is a potential extraction surface. The categories at risk: trade secret instructions embedded in system prompts, moderation and safety rules that competitors could extract and bypass, and specialized fine-tuned instructions that represent significant development investment.

The research parallels work on reasoning trace extraction — a pattern of academic work systematically demonstrating that information companies assume is hidden inside AI systems is more recoverable than expected.

What This Means for Builders

Three practical responses are worth evaluating now:

Output perturbation — Introducing controlled variation in model outputs (paraphrasing, synonym substitution) to increase reconstruction difficulty. This degrades PTP accuracy without changing the user experience substantially.

Prompt segmentation — Distributing sensitive instructions across multiple API calls rather than a single system prompt, reducing the signal available from any single output sample.

Access-controlled instrumentation — Treating system prompt content as a credential, not a configuration value. Logging access, detecting unusual query patterns that might indicate systematic extraction attempts, and rotating high-value prompt content on a schedule.

The defense posture here mirrors enterprise AI security frameworks that treat AI system configuration as part of the security perimeter — not ancillary to it.

Source: The Decoder

Content created with AI assistance and reviewed for accuracy.