Zero-Mem: Zero-Token Memory Operations for LLM Agents | AI Paper Digest

TL;DR Highlight

Zero-Mem cuts LLM calls from agent memory operations, reducing memory time 57.6% versus the fastest baseline.

Who Should Read

A backend developer building long-term agent memory who is worried about the cost and latency of LLM calls used for summarizing and storing memory.

Core Mechanics

Memory operations use no LLM
- Only the final-answer reader calls an LLM.
- No other step consumes LLM input or output tokens.
- Encoder computation is accounted for separately.
Keep original traces, not summaries
- Original interaction traces are the source of record.
- LLM-made intermediate records can omit or merge details and obscure the evidence.
Two views organize the same traces
- An entity-context graph exposes connections across interactions.
- A temporal hierarchy preserves conversational locality and session state.
Each query weighs the two views
- It retrieves from both views using query-dependent weights.
- It follows their structure to recover supporting relations or surrounding context.
Rules discard conflicting evidence

Community Reactions

Ablations back both views
- Ablations reportedly support the contribution of the two views and their query-dependent coordination.
Experience with a KV-cache approach
- A commenter stores conversation KV cache in a local LLM and retrieves via attention scores after a prefill pass.
- They report SOTA results on LoCoMo and LongMemEval.
Opinion that it is easy to implement
- One commenter says it is easy to build at the agent harness level.
- They say NER can be much simpler for programming and long-running task dialogues.
Opinion that the real win is no generative rewriting
- A commenter says summaries make retrieval depend on omissions, hurting auditability.
- They want benchmarks on contradiction and changing values, and encoder and index costs reported next to token cost.

How to Apply

Count LLM calls in your memory pipeline
- Log call count, tokens, and latency for each summarize, rewrite, and retrieval step.
- Check what share of calls happens outside the final answer.
Store raw turn IDs next to summaries
- If you use summaries, attach source turn IDs to each one.
- Show the raw turns that supported each answer.
Try retrieval with entity and session indexes
- Build an entity index and a session/time index, and query both with the same question.
- Compare which works better per question type.
Build a conflict test with changing values
- Create conversations where attributes change across sessions and check if latest and earlier values are distinguished.
- Record encoder and index costs alongside token costs.

Try It

copy & run

# Simplified version of the paper's method: memory audit prompt to paste into Claude/ChatGPT

You are a reviewer of agent memory design. In the memory pipeline below, separate steps that need an LLM call from steps that rules or indexes can replace.

[Pipeline description]
{{memory write/retrieval flow}}

[Sample conversation log]
{{10-20 turns of conversation}}

Instructions:
1. Make a table of each step, whether it calls an LLM, and whether it consumes input/output tokens.
2. Assume the raw conversation is kept with an entity-based index and a time/session-based index, and design the fields for each index.
3. For the question {{question}}, write rules for how much to use each index and how to discard conflicting evidence.

Next to try: feed in a conversation where a value changes (e.g., address or job title) and ask which is the latest and which is earlier.

Terminology

zero-token memory operationMemory writes and retrievals that never call an LLM. Only the final answer generation uses one.

entity-context graphA map with entities such as people and things as nodes, linked to where they were mentioned. It helps find connections across scattered conversations.

temporal hierarchyConversations grouped by session and turn order, like a folder tree. It keeps surrounding context and session state.

deterministic calibrationA rule-based step that drops conflicting evidence and ties the answer to the evidence, without an LLM.

ablationAn experiment that removes components one at a time to see how much each contributes.

Related Papers

Related Resources

https://arxiv.org/abs/2607.29377