I accidentally turned LLM memory into program analysis

TL;DR Highlight

Keeping LLM memory as Datalog rules beat all published systems on knowledge-update questions

Who Should Read

Agent developers frustrated that their LLM keeps reasoning from assumptions already disproved during long sessions

Core Mechanics

Memory is two problems: retrieval and current truth
- Vector DBs find past notes relevant to a question.
- They don't know if a note was later disproven or if conclusions built on it are now invalid.
The LLM extracts facts, the database reasons
- The LLM turns logs, debugger output and prose into structured facts.
- A Datalog engine (Lemmalog) derives and maintains conclusions from rules.
Removing a fact auto-invalidates its conclusions
- It tracks which facts support each conclusion, keeping it if another derivation remains.
- With no support left, the conclusion disappears.
You can ask why it believes something
- Every conclusion carries provenance (which observations and rules produced it).
- A conclusion without provenance isn't part of the maintained state.
Facts get validity intervals so history survives

Community Reactions

It led on knowledge updates
- On LongMemEval Knowledge Update, Lemmalog scored 0.579 vs PropMem 0.528 and Full Context 0.202.
- Multi-Session was weak at 0.211 vs PropMem 0.582.
On full LoCoMo it ranks third
- Three runs gave F1 0.533±0.001, behind PropMem 0.605 and OpenClaw 0.557, and Full Context at 0.542.
- Adversarial false-premise questions scored 0.707 vs 0.509 for Full Context.
Failures were extraction misses, not reasoning
- Multi-session failures were mostly facts never extracted rather than mis-connected.
- Extraction used Claude Sonnet 4.6; later stages used the benchmark's standard readers and judges.
Commenters recognized the stale-facts problem
- One commenter said Claude records things everywhere and treats disproved facts as true.
- Another said an LLM kept resurfacing a ruled-out cause in a hardware failure investigation.

How to Apply

Add a retraction list to your agent notes
- Today, add 'rejected assumptions' and 'conclusions depending on them' sections to CLAUDE.md or your notes.
- Instruct the agent to remove or mark invalid any dependent conclusion when an assumption is overturned.
Split memory into retrieval and current state
- Keep your vector memory and add a separate store holding only currently-true facts.
- Have answers prefer the current state and use retrieval as support.
Require evidence IDs on every conclusion
- Make supporting observation IDs a mandatory field when the agent writes a conclusion.
- Add a validation step that rejects unsupported conclusions.
Check whether your grounding instruction causes refusals
- If you added anti-hallucination prompts, measure refusal rate on answerable questions.
- Write: refuse if the premise is absent, but reason if evidence exists and needs counting, comparing or ordering.

Try It

copy & run

# Paste at the start of a Claude or ChatGPT chat (simplified version of the post's method: mimics Datalog via prompt)

You are a research assistant. Maintain a 'maintained state' in this format throughout the conversation.
- FACTS: numbered observations (F1, F2...). Each marked valid/retracted
- CONCLUSIONS: numbered (C1...). Each must list its supporting FACT numbers
- Rule: when a FACT is retracted, immediately mark invalid every CONCLUSION that depends only on it. Keep conclusions that still have other valid support.
- Never write a conclusion without supporting fact numbers.
- End every reply with the updated state.

Topic: {{research topic}}

I'll now give observations; update the state. First observation: {{observation 1}}

Next to try: later say '{{observation 1}} was wrong' and ask which conclusions became invalid and why.

Terminology

DatalogA declarative language where you state facts and rules and it derives new facts automatically. Like spreadsheet formulas, outputs update when inputs change.

ProvenanceA record of which facts and rules a conclusion came from. Think git blame for conclusions.

RetractionRemoving a fact already added. Conclusions with other support stay; those that lose all support disappear.

Incremental evaluationRecomputing only the parts affected by a changed input, like a build system recompiling only changed files.

LongMemEval / LoCoMoBenchmarks that evaluate LLM memory systems with questions about long conversation histories.