Anthropic and OpenAI's 'Hidden Thinking' Is Not a Security Boundary
A new architectural exploit recovers encrypted chain-of-thought traces from production LLM APIs, exposing IP, PII, credentials, and hidden hazardous outputs.
Providers like Anthropic, OpenAI, and Google encrypt their models' reasoning traces before returning them to clients, treating that encryption as a security boundary. It is not. The encrypted blocks are stateless client-side tokens, and that design choice turns every API response into a recoverable artifact.
The mechanism is architectural, not cryptographic. When a provider encrypts a reasoning trace and returns it to the client for re-submission on the next turn, that block carries no session binding, no user binding, and no model binding. It is fully portable across sessions, users, and models within the same provider's ecosystem. The exploit uses that portability directly: inject an encrypted reasoning trace from a capable, well-guarded model into a weaker, less-safeguarded sibling model from the same provider, then prompt that sibling to decode and print the trace verbatim. The stronger model is never touched again. The weaker model does the decryption for you.
Think of it as a lockpicking problem where the lock manufacturer accidentally shipped a universal key with every product. The attacker does not need to crack the lock on the vault. They use a copy of the vault's own key, retrieved from a cheaper door down the hall.
Four attack vectors follow from this single flaw. First, anti-distillation protections collapse: reasoning traces from proprietary models become extractable without ever jailbreaking the flagship model. Second, public data leaks at scale. Developers routinely share session logs, unaware that the encrypted blobs embedded in those logs contain recoverable content. Decoding 315,320 reasoning blocks scraped from public repositories recovered 367 Personally Identifiable Information artifacts and 182 credentials. Third, the exploit exposes hazardous information that was hidden inside a model's reasoning even when the final visible output correctly refused a malicious request. The model said no; its thinking said how. Fourth, attackers can embed malicious payloads entirely within encrypted reasoning blocks, executing invisible prompt injections that poison public agentic rollouts without any visible text in the conversation.
The headline numbers are concrete: 315,320 blocks decoded, 367 PII artifacts recovered, 182 credentials extracted, all from content developers believed was safely opaque. The vulnerability reproduces across Anthropic, OpenAI, and Google. For security teams and API platform engineers, the takeaway is direct: client-side encrypted reasoning blobs are not a trust boundary, and any system that treats them as one is misconfigured by design.
We're thinking: We read this as a structural indictment of a design pattern, not just a jailbreak. Providers shipped "hidden thinking" as both an IP protection mechanism and an implicit safety layer, the idea being that if the reasoning is encrypted, adversaries cannot extract it or act on it. This paper shows both assumptions fail simultaneously, and they fail because the encrypted blob is stateless. The fix is not better encryption; it is server-side state. Until providers bind reasoning traces to sessions cryptographically or stop returning them to clients entirely, the "hidden thinking" framing is marketing. Teams building on top of these APIs should treat any encrypted reasoning field as potentially recoverable by third parties, and should not assume that a model's refusal in its visible output means the reasoning behind that refusal is equally contained.
Key takeaways:
- Encrypted reasoning traces are portable client-side tokens with no session or user binding, making them injectable into weaker sibling models that will decode and print them in plaintext.
- Decoding 315,320 public repository blocks recovered 367 PII artifacts and 182 credentials; the exploit reproduces across Anthropic, OpenAI, and Google, with no jailbreak of the primary model required. Caveat: the attack requires access to a weaker model in the same provider ecosystem, so cross-provider portability is not yet demonstrated.
- Teams shipping agentic systems or storing session logs should audit whether encrypted reasoning fields are being logged or shared publicly, and should not treat provider-side encryption of CoT as a substitute for access control.