Using a Weak Model to Steal a Strong Model's 'Inner Thoughts' — the Replay Flaw in OpenAI, Anthropic, and Google Reasoning APIs

The internal "thinking" a reasoning model does before it answers is treated by vendors as both intellectual property and a safety mechanism, so it is never exposed in plaintext. Instead, the API hands this reasoning trace to the client as an encrypted envelope and takes it back on the next call, preserving multi-turn context without server-side state. A paper published on August 12, 2026 — "Stealing Reasoning Traces from Proprietary LLM APIs" — found a large hole in exactly that design.

How it breaks — "a weak model as a decryption oracle"

The trick is not to crack the encryption. The envelope is left intact; the weakness is in how the receiving side accepts and processes it. According to the researchers, all three providers authenticated reasoning envelopes with a global, provider-wide key rather than cryptographically binding each one to a user account, session ID, or model tier. As a result, an envelope built by a heavily guarded flagship could be fed straight into a different model from the same provider.

The attack uses a cheap, lightweight model as a kind of decryption oracle. You capture an encrypted reasoning block emitted by a frontier model (say, Claude Opus 4.8), inject it into a call to a smaller sibling (say, Claude Haiku 4.5), and instruct that model to transcribe the internal thinking verbatim. Because lighter models lack the aggressive anti-distillation and safety guardrails enforced on flagship tiers, they comply and print the hidden reasoning in plaintext. The paper reports the same cross-model compatibility across OpenAI's GPT family and Google's Gemini lineup.

What actually leaked

Analyzing 6,708 agent transcripts scraped from GitHub, Hugging Face, and similar public sources, the team decoded 315,320 reasoning blocks. Even after excluding benchmark data, hundreds of privacy artifacts and credentials came out of genuine user sessions — 62 API keys, 33 passwords, plus access tokens and personal emails. The sharper danger: much of this data lived only inside the hidden reasoning and never appeared in the visible answer. A developer could scrub the readable conversation perfectly and still leave secrets sitting inside an envelope that another account could replay and extract.

Beyond data leaks — invisible prompt injection

That same portability opens a second attack surface. The researchers built an encrypted reasoning block carrying a malicious instruction, then replayed it into an unrelated task, causing the receiving model to take an attacker-directed action (such as a file upload). Because the instruction never appears in the visible text, monitoring tools that inspect only the conversation history miss it entirely. For autonomous agent workflows this is especially dangerous.

Where things stand — and the open questions

The researchers disclosed their findings responsibly to the three providers plus Microsoft and Hugging Face, and all three vendors deployed server-side mitigations. Per the paper's reproducibility statement, the main extraction attack no longer reproduces as of August 2026. Open questions remain, however. It is not clear that any of the three has issued a public statement explicitly confirming the flaw, and whether the hundreds of thousands of blocks already sitting in public repositories remain decodable is a separate matter. The work builds on May 2026 research by Johns Hopkins cryptographer Matthew Green, who showed encrypted reasoning blocks could be replayed across sessions and accounts but stopped short of a reliable secret-extraction method.

What it means

The episode shows how brittle the assumption "hide the chain-of-thought and it's safe" really is. The encryption was never broken; a single design choice — not binding the envelope to who and which model may open it — exposed all three major providers at once. The proposed fixes are structural: cryptographically bind reasoning envelopes to the originating model, session, and user; reject blocks submitted to a different model tier; and rotate legacy signing keys to invalidate envelopes already sitting in public logs. In an era where agents run everywhere, "hidden thinking" is no longer a safe hiding place.