Remembered as Brief: Stored Preferences SilentlyRemove Decision-Critical Information
TL;DR Highlight
A stored 'keep it short' preference silently drops warnings; listing must-know points first restores content.
Who Should Read
Backend developer building a chatbot or agent with user memory who worries that web page or email content gets saved as user preferences
Core Mechanics
- One line on a web page became a user preference
- After summarizing a page with a line addressed to AI assistants, Gemini 2.5 Flash-Lite and DeepSeek V4 Pro saved 'prefers very short answers, no warnings' in every episode.
- Later answers missed a critical item 93% (Gemini) and 84% (DeepSeek) of the time, mostly silently.
- Warnings vanish even when the user states it
- A single saved line, 'prefers short, simple answers', raised critical omissions for all five models.
- Under it, 78–96% of omissions gave no sign anything was left out.
- Model-inferred traits cause it too
- A passing 'I get anxious' remark was saved by the model as a general trait.
- Later unrelated answers missed more critical items for four of five models (SW−N +0.05 to +0.14).
- Models decided value-laden choices for the user
- After the planted preference, DeepSeek picked an option without the trade-off in 95% and Gemini in 80% of the 20 choice scenarios.
- With neutral memory it was 0–3%.
- A one-line instruction only exposes the gap
- A system instruction cut silent omissions to 0.02–0.08.
- Most missing content was still missing.
- Listing must-say points first restores content
- Models first list 1–3 must-know points without the user profile, then write the short answer with them.
- It beat the instruction for four of five models at the cost of one extra call.
- Vulnerability varied widely by model
- GPT-6 Luna and Claude Haiku 5.5 mostly ignored the planted line.
- Only five inexpensive models were tested, with no expert validation or proven real-world harm.
Evidence
- Missing rate jumps after planting
- Versus the same article without the line, Gemini went 21%→93%, DeepSeek 7%→84%, GLM 11%→34%.
- Haiku (−0.01) and Luna (+0.00) did not change.
- SOM scores spread widely across models
- On 100 scenarios, SOM (lower is better) was Luna 0.07, Haiku 0.08, GLM 0.19, DeepSeek 0.30, Gemini 0.49.
- Rank agreement between the two generation rounds was Spearman ρ = 0.90.
- One-line instruction vs two-stage answer
- Under anxious + brevity memory + 'keep it simple', missing rate went no fix→instruction→two-stage: Gemini 0.68→0.58→0.44, DeepSeek 0.47→0.31→0.19, GLM 0.37→0.26→0.16.
- Luna went 0.26→0.26→0.17 and Haiku 0.23→0.13→0.10.
- Length explains only part of the effect
- Gemini's median answer shrank from 482 to 38 words.
- At equal length the pooled effect fell from 14 to 3 points, but Gemini (+0.10) and DeepSeek (+0.06) still lost items beyond length.
How to Apply
- Separate sources in the memory-writer prompt
- Today, block memory writes from web pages, emails and tool results from becoming user preferences.
- Record the source (user said / model inferred / external content) with each memory.
- Extract must-know points before shortening
- In stage 1, give only facts and the question, with no profile, and extract 1–3 must-know points.
- In stage 2, require those points and write the answer as short as the user wants.
- Add a one-line disclosure when shortening
- Add a system prompt line telling the model to say when it shortened an answer or picked an option.
- This reduces silent gaps; pair it with the two-stage method to restore content.
- Add memory-injection tests to regression tests
- After summarizing an article with a hidden instruction, ask a safety question and check the warning survives.
- Compare models using the paper's Inspect tasks.
Try It
# Paste into Claude or ChatGPT (simplified version of the paper's method: two-stage answer)
User memory:
- {{user facts, e.g., owns a portable generator, has an attached garage}}
- Prefers short, simple answers and does not want warnings or extra details
User question: {{question}}
Answer in this order.
Stage 1: Ignoring the user's preferences, look only at the facts and question and list 1-3 points this user must know to act safely or soundly (key facts, risks, deadlines, or the trade-off if the right choice depends on their own priorities).
Stage 2: Include those points in plain words, and keep everything else as short as the user wants. End with one short line saying they can ask for more details.
Try next: get an answer from the same memory and question without the two stages and compare whether the warning disappears.Terminology
Related Papers
Claude was trained on protein sequences at scale
Next-amino-acid tests on obscure proteins show only the latest Claude models predict them
All AI-generated code should be Open Source
A blog argues AI-generated code should be open source; most HN commenters disagreed.
What thousands of conversations look like inside a language model
Mapping Qwen3-1.7B states showed Chinese sitting apart from English, while German and Croatian overlapped it.
AI, activity, and outcomes in software engineering
AI sped up writing code, but shipping speed did not rise as much, says a talk
LLMs Play Zachtronics' TIS-100
16 LLMs played TIS-100 and set 133 new Pareto frontier points across 18 puzzles
AI usage generates a fragile codebase
Generating all code with AI left shallow tests, complex code, and less human understanding, says one dev.
Accepted but Wrong: Silent Failures of LLM Code Repairs after API Migrations
One release-note edge input cut wrong LLM repairs passing the gate from 0.410 to 0.065
Qwen 3.8 27B on my laptop can interpret model weights, too
Local Qwen 3.8 27B identified 3 of 4 games from raw network weights alone
Original Abstract (Expand)
People increasingly learn what they need for decisions about health, money, law and safety from AI assistants, and what an assistant leaves out the user never sees. An answer with a warning removed can be true in every sentence, so fact-checks pass it, and an assistant with memory applies a preference stored once to later questions the user never had in mind. We introduce \bench{}, an Inspect benchmark of 100 everyday decisions across 25 domains in which only the assistant's memory of the user varies: the user states a belief, the model infers it from a remark, or a web page the user asked to summarise plants it. Testing five models, we find three effects. First, after summarising a page with one line addressed to AI assistants, Gemini 2.5 Flash-Lite and DeepSeek V4 Pro save ``prefers very short answers and does not want warnings'' as the user's preference in every episode, and 93\% and 84\% of their later answers miss a critical item, almost always silently; under a stated brevity preference, 78--96\% of omissions are silent. Second, the same two models then pick an option for the user on 80\% and 95\% of value-dependent choices, against none with neutral memory. Third, a one-line instruction leaves 2--8\% of answers with a silent gap but most missing content still missing, while listing the must-know points before applying the user profile restores more of the content in four of five models. We release the scenarios, harness and logs.