Stealing Reasoning Traces from Proprietary LLM APIs | AI Paper Digest

TL;DR Highlight

Replaying encrypted reasoning blocks into a weaker model decoded them to plaintext, leaking keys and PII.

Who Should Read

Backend and agent developers who store or publish agent logs containing Claude, GPT, or Gemini thinking blocks.

Core Mechanics

Encrypted reasoning blocks are interchangeable
- Anthropic, OpenAI, and Google return reasoning as encrypted blocks that the client sends back on the next request.
- These blocks work across sessions, users, and models within the same provider.
- So a weaker, less safeguarded model can act as a decryption oracle.
Haiku transcribes Opus's hidden thoughts
- Send an Opus 4.8 thinking signature to Haiku 4.5 and ask it to copy the reasoning verbatim, and it outputs the plaintext.
- The stronger model never has to be jailbroken directly.
- Haiku needed one fixed prompt, while GPT-5.6 Luna needed varied templates and best-of-n.
Public agent logs leak secrets
- Users can't read the encrypted blocks, so they can't sanitize them before sharing.
- Asking an agent to clean up a session made it restate sensitive values in hidden reasoning.
- Some PII never appeared in the visible chat at all.
Harmful info leaks via the reasoning channel

Evidence

Large-scale decoding of public traces
- From 6,708 agent trajectories on GitHub and Hugging Face, they recovered 315,320 reasoning blocks.
- 0.3% (1,028) of blocks leaked private data, and 4.9% (328) of sessions leaked at least one item.
Volume of recovered sensitive data
- They recovered 367 PII artifacts and 182 credentials.
- Genuine user sessions alone yielded 62 API keys, 33 passwords, and 30 personal emails.
Values absent from visible chat
- Of 704 artifacts recovered from genuine user sessions, 64 were completely absent from the visible chat history.
Cost and fidelity
- Decoding 10k traces with 12k-token input and output at Claude Haiku 4.5 pricing costs about $720.
- On 120 Codeforces problems, API-reported thinking tokens and reconstructed token counts mostly tracked each other closely.
- The whole study spent roughly $30,000 in API credits.

How to Apply

Strip thinking blocks from logs and traces
- Before pushing agent logs to GitHub, HF, or shared workspaces, remove signature, thinkingSignature, and encrypted_content fields.
- Plaintext masking alone isn't enough, so grep your repos today.
Audit logs you already published
- If public repos or datasets still contain signature fields, take them down and rotate any keys that may have been exposed.
Don't treat reasoning blocks as secret storage
- If your agent handles PII or keys, put only needed values in context and avoid asking it to "clean up" a session in a way that makes it restate them.
Remove signed blocks before resuming shared sessions
- When continuing someone else's agent log, drop the reasoning blocks first.
- This guards against hidden injections.

Try It

copy & run

# Paste into Claude Code or a terminal (simple version of the paper's hygiene advice: detect reasoning signature fields in logs)

# Run from the log folder
grep -rlE '"(signature|thinkingSignature|encrypted_content)"' ./logs

# Removal example before publishing (jq)
# jq 'walk(if type=="object" then del(.signature, .thinkingSignature, .encrypted_content) else . end)' in.json > out.json

# Next: grep again to confirm nothing remains, and rotate any keys that were in already-published logs

Terminology

Chain-of-thought (CoT)The scratch-work a model writes before answering. It holds far more information than the final answer.

Encrypted reasoning blockAn opaque string holding hidden reasoning that the client carries and sends back on the next request. Like carrying a locked box without the key.

AEAD envelopeA wrapper that both encrypts and authenticates data. It holds a header, nonce, tag, and ciphertext.

Decryption oracleSomething that turns ciphertext into plaintext on request. Here, a compliant weaker model plays that role.

DistillationTraining a weaker model on a stronger model's outputs. Including the reasoning makes it much more effective.

PrefillPre-filling the beginning of a model's response or reasoning so it continues from there.

Related Papers

Related Resources

https://stolen-thoughts.com

Original Abstract (Expand)

Leading large language model providers now conceal their models' step-by-step reasoning, or chain-of-thought, to protect intellectual property and limit information leakage. Rather than storing these traces server-side, providers return them to the client as blocks of encrypted text, which the client passes back with each subsequent request. Building on prior research, we identify an architectural vulnerability: these encrypted blocks are fully compatible and interchangeable across different sessions, users, and models within a provider's ecosystem. We exploit this compatibility to develop a scalable decryption jailbreak. By injecting an encrypted reasoning trace from a given model into a weaker, and less safeguarded model from the same provider, we force it to decode and output the trace verbatim in plaintext, without ever jailbreaking the more capable model directly. This vulnerability enables four distinct attack vectors. First, it circumvents anti-distillation mechanisms, allowing adversaries to extract a proprietary model's reasoning, as we demonstrate across Anthropic, OpenAI, and Google. Second, it allows for large-scale private data extraction. Developers frequently share session logs publicly, unaware of contents of the encrypted blocks. By decoding 315,320 reasoning blocks scraped from public repositories, we recovered 367 Personally Identifiable Information (PII) artifacts and 182 credentials. Third, it inadvertently reveals hazardous information hidden within the reasoning process, even in cases where the model's final, visible output safely rejects a malicious request. Fourth, attackers can leverage this flaw to execute invisible prompt injections, embedding malicious payloads entirely within encrypted blocks to poison public agentic rollouts. Following responsible disclosure, we propose concrete cryptographic and system-level mitigations to secure client-side reasoning.