Presentation, Not Mechanism: A Render Confound in Deprecation-Aware Memory Evaluation

Who Should Read

A backend developer deciding whether to add structured memory (typed edges, deprecation flags) to an agent or RAG system.

Core Mechanics

• Most of the gain was formatting
  • On reverted-value questions (30s→60s→30s), RevisionLedger beat a flat baseline (GraphRAG+abstain) by +0.182.
  • A control with the same layout but deprecation switched off removed almost all of that gain.
  • A readable, time-ordered table helped the answer LLM, not the memory logic.
• Fine-grained typing is idle for current-value queries
  • Once layout was matched, the fine mechanism contributed +0.025, indistinguishable from zero.
  • This fits the theory that a binary live/dead flag is enough for current-state answers.
• Coarse invalidation wins on current-state queries
  • Zep/Graphiti-style edge invalidation beat the fine ledger by 0.084.
  • Injecting relation or extraction noise never made the fine ledger win.
• For history queries, retention is what matters
  • A store that discards invalidated edges scored 0.243 on 'what was the previous value?'.
  • Keeping edges with validity windows raised it to 0.604.

Evidence

• Render decomposition (DeepSeek-V3 judge)
  • The reverted-revert gain of +0.184 splits into a render term of +0.159 and a mechanism residual of +0.025 [-0.005, +0.057].
  • An out-of-family MiniMax-M2.5 judge gives render +0.164 and fine mechanism +0.021.
  • Coarse invalidation's mechanism contribution is +0.087 [+0.046, +0.130].
• Coarse vs fine (reverted-revert, n=439)
  • Graphiti beat GraphRAG+abstain by +0.266 and beat RevisionLedger by 0.084 (RevisionLedger minus Graphiti = -0.084 [-0.128, -0.041]).
  • In noise sweeps (n=445/level), the gap at p=0 was -0.016 (n.s.), and coarse led or tied at every noise level.
• Provenance queries (979 queries, n=4,895)
  • Discarding coarse store 0.243, d-blind 0.439, retaining coarse 0.604, fine 0.715.
  • The fine vs retaining-coarse gap is +0.111.
  • On external TempLAMA, retention gave +0.514.
• Relation-reading queries
  • On REFINES, a learned scorer gave +0.147 vs GraphRAG+abstain, versus -0.086 with the rule scorer.

How to Apply

• Add a render-matched control to your memory A/B test
  • When comparing structured memory with flat RAG, add a condition with the same table layout but deprecation marked 'neutral'.
  • Compare accuracy across all three conditions with the same questions and judge.
• Try turning flat context into a time-ordered table
  • Sort events into columns for time, evidence, and value, and put that in the prompt.
  • Measure accuracy before and after on your existing eval set today.
• For current-value queries, implement only a binary live/dead flag
  • When a new event supersedes an old one, stamp invalid_at on the old item.
  • Skip building a relation-type classifier.
• If you have history queries, never delete invalidated items
  • For 'what was the previous value?' queries, store invalidated edges with their validity windows.
  • If privacy is a concern, set a retention cap and propagate deletions.

Try It

# Paste into Claude or ChatGPT (simplified version of the paper's method: render-matched comparison)

From the event stream below, answer 'what is the value currently in force?'. Answer in three ways and compare them.

[Events]
{{event list: time, content}}

[Question]
{{question, e.g. what timeout is currently in force?}}

Method A (flat): read the events as unordered plain text and answer.
Method B (render only): arrange them in a time-ordered table (time | evidence | value | status) with every status set to neutral, then answer.
Method C (deprecation marks): use the same table, mark superseded items DEPRECATED and valid ones LIVE, then answer.

Finally, summarize how the answers differ between A and B, and between B and C.

Next, change the question to 'what was the previous value?' and compare a table with the DEPRECATED rows removed against one that keeps them.

Terminology

Related Papers

• The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge

  Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.

• 90% of US working hours are spent on tasks not covered well by AI benchmarks

  Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage

• Postgres 19 delayed after AI reviewers find more bugs than humans can fix

  LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted

• Show HN: Gemini 4 Argon plays StarCraft

  LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.

• A Plausible Explanation Is Enough to Mislead Multimodal Judges

  Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.

• Neural networks are clean

  Looking at neurons directly shows most have one main job, challenging superposition.

Related Resources

• https://github.com/Anonymous-Awesome-Submissions/ESR-pipeline
• https://github.com/Anonymous-Awesome-Submissions/ESR-bench