Memorilla: Latent Semantic Memory for LLMs

Who Should Read

A backend developer building agents or RAG who struggles with token cost and accuracy from stuffing long chat histories or documents into context.

Core Mechanics

• Documents become a few tokens
  • Memorilla gathers a whole document collection into a fixed number of query-dependent memory tokens for a frozen LLM (model with fixed weights).
  • Documents are encoded once into reusable embeddings (vectors representing text).
• Injected into input embeddings, not as text
  • It learns to retrieve query-specific information and place it in the LLM's input embedding space.
  • It does not paste retrieved text into the prompt.
• Best on 5 of 8 benchmarks with 16 tokens
  • Baselines include Mem0 and MemoRAG.
  • Benchmarks span personalization, long-document and knowledge-intensive QA.
• Beat full context on FactKG
  • Accuracy was 84.33% versus 80.43% for full-context inference.
  • It used 108x fewer tokens to represent the source documents.
• Higher agent success in TextWorld
  • Average success rate rose from 18.6% with full-history in-context learning to 24.2%.
  • The agent wandered less between rooms it had already explored.

Community Reactions

• Memory tokens inject a vibe, one commenter says
  • The reading is a specialized RAG embedding that injects responses directly into the main model's input stream without text.
  • That fits the reported strength on concepts and weakness on quotes and exact text.
• Pushback: full-text search is enough
  • One commenter finds semantic and lexical search nearly equivalent when a frontier model drives.
  • They say models write Lucene and other FTS queries well enough to get semantic-like results.

How to Apply

• Separate exact-quote questions in your RAG
  • Spend 30 minutes today counting conceptual versus exact-wording questions in your query logs.
  • Route exact-wording questions to raw-text retrieval instead of compressed memory.
• Build a full-context vs retrieval eval set
  • For a feature with long chat history, compare accuracy and token count between full context and your current memory approach on the same questions.
  • Use that gap to decide whether to adopt compression.
• Use FTS as a baseline
  • Before adding embedding pipelines, measure accuracy when the model generates FTS queries.
  • For tasks with long agent histories, measure against Mem0 as well.

Try It

# Paste into a Claude or ChatGPT chat (simplified version of the paper's method: mimics query-specific compressed memory without training)

You are a memory compressor. From the document below, pick only the information needed for the question and compress it into at most 5 short unnumbered notes. Keep exact quotes, numbers, and proper nouns verbatim.

[Document]
{{문서}}

[Question]
{{질문}}

Output the compressed notes first. Then answer the question using only those notes, and write 'insufficient memory' for anything the notes cannot answer.

Next step: ask the same question with the full document and compare answers and token counts.

Terminology

Related Papers

• What the Author Wrote, What the Agent Reads: Silent Cue Loss in Enterprise Document Ingestion

  Preserving red text and highlights in docx ingestion lifted marked-requirement F1 from 0.150 to 0.690.

• Multimodal Reranking Is a System One Decision

  One multiple-choice question to a VLM over 16 candidates reranks them, beating a dedicated 2B reranker.

• You could have invented transformers from memory

  Viewing linear, MLP and attention layers as memory stores: capacity goes ~64 to ~2,700 to over a million

• Pointing AI at archives found a forgotten meteorite, lost rhinos, and more

  Embedding search plus a cheap filter model surfaced unrecorded events in 400 years of archives

• EmbeddingGemma 2: An open, lightweight multimodal embedding model

  One open 740M model embeds text, images, audio and video into a single shared vector space

• RAG-NAROK: Retrieval-Aware Knowledge Corpus Poisoning in RAG with Source-specific Refutation

  Refutation documents targeting retrieved sources took top rank and evaded anomaly detectors up to 88.37%.

• Text, Pixels, or Both? Evaluating Input Representations for Multimodal Document QA

  Page images beat OCR text on accuracy but cost more; a question-only router cut latency 30%.

• Microsoft exec called AI scraping 'the largest theft of labor in human history'

  NYT lawsuit unredacted filings reveal Microsoft and OpenAI internal views on scraping

Related Resources

• https://arxiv.org/abs/2610.21375