Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions

Oct 27, 2026•Anouar Imel, Imad Charradi, Abderrahman Sghir +1•View PDF

TL;DR Highlight

Picking group-discussion memories by rarity or relevance often flips the final LLM decision

Who Should Read

A backend developer building a multi-agent pipeline that compresses discussions into memory for a final decision model

Core Mechanics

Rarest-first equals random on average
- Keeping the rarest claims (R1) matched Random on average across CrossProfile.
- It wins only when rare facts happen to be the supporting ones and loses when they mislead.
Relevance filters readmit decoys
- Decoys are true, on-topic statements, so a relevance filter keeps them.
- Even a perfect filter scored below R1 in the classic hidden-profile cell.
Balance of kept evidence drives accuracy
- Accuracy rose monotonically with the correct option's net evidence lead among kept facts.
- Once that lead is known, the selection policy adds little.
Many-agent tags earn extra trust (social proof)
- The decision model weighted facts tagged as stated by many agents more heavily.
- In the tasks those facts were assigned to everyone, not independently observed.
Natural discussions lose evidence before memory
- The full transcript scored below an empty memory.
- An evidence-first prompt helped Llama a lot but did nothing detectable for Qwen.

Evidence

Same as random on average
- CrossProfile mean accuracy: R1 64.0% vs Random 64.5% (Qwen), R1 63.0% vs Random 63.4% (Llama).
- With private support and shared decoys, R1 hit 90–97% and B4 (first-mention) 11–31%; in the reverse layout B4 hit 92–96% and R1 18–40%.
Even a perfect relevance filter lost
- In the classic cell, oracle relevance minus R1 was −23.3p (Qwen), −31.1p (Llama), −29.4p (DeepSeek-V4.1-Flash).
- R1-rel minus R1 was −22.2p (Qwen), −20.0p (Llama), −32.2p (Flash).
Removing tags raised accuracy
- In the classic cell with all-facts memory, dropping tags gave +14.4p (Qwen), +13.3p (Llama), +24.4p (Flash; 72.8%→97.2%).
- Randomly reassigning tags with line order fixed gave +12.8p on both open models.
Transcript worse than empty memory
- Natural HiddenBench REF accuracy: Qwen 22.1%, Llama 33.8%, Flash 25.6%, vs empty memory 33.8%, 36.9%, 35.9%.
- With all facts given: Qwen 70.3%, Llama 61.5%, Flash 89.7%.
Evidence-first helped only Llama
- Llama REF accuracy rose +30.8p, and +25.1p in a paired ablation with fresh seeds.
- Qwen showed no detectable change.

How to Apply

Try stripping agent tags from the memory
- Compare your decision model's accuracy with and without speaker tags like [A1,A2] in the memory.
- If sources aren't independent, make the untagged version your default.
Evaluate summary policies by evidence balance
- On your eval set, count pro and con facts kept by each policy and log the correct option's net lead.
- Separate accuracy differences caused by the policy from those caused by what was kept.
Add evidence-first to the discussion prompt
- Tell agents in their first turn to state concrete observations with conditions, negations and numbers before recommending.
- Measure full-transcript accuracy on your own model, since effects differ by model.
Test memory policies on ownership-varied cases
- Build cases where the same facts are shared vs. private across agents and run each policy.
- Check per-cell accuracy, not just the average.

Try It

copy & run

# Paste into Claude or ChatGPT as-is (simplified version of the paper's method: compare decisions with and without tags)

Decide the {{question}} below using two memories separately. Memory A has speaker tags and Memory B does not. Use only each memory as evidence, and answer in the format 'Reasoning: (at most five sentences) / Decision: (option copied exactly)'.

Question: {{question}}
Options: {{options}}

Memory A (tagged):
{{memory like [A1,A2,A3,A4] fact1 / [A2] fact2 ...}}

Memory B (untagged):
{{same facts with the tags removed}}

Next step: if the two decisions differ, ask which facts it weighted more and why.

Terminology

hidden-profile taskA decision task where each participant holds only some facts, so the group must pool information to get the right answer, like assembling a puzzle from pieces split among people.

decoy factA true, criterion-relevant fact that favors a competing option and misleads the decision.

R1 (rarest-first)A memory policy that keeps claims in order of fewest agents who stated them.

CrossProfileA diagnostic benchmark of 60 templates (480 tasks) that crosses shared/private ownership of support, decoy and irrelevant facts.

social proofTrusting information more because many people said it; here, the decision model trusting facts tagged with many agents.

Related Papers

Original Abstract (Expand)

When an LLM group's discussion is compressed into a short memory for a final decision maker, what should the memory keep? Two natural heuristics keep the rarest claims or the claims relevant to the decision criteria. On hidden-profile tasks both look attractive, because how widely a fact is held is confounded with whether it helps. We introduce CrossProfile, a diagnostic benchmark of 60 templates with automatically verified answer keys, in which the ownership of supporting, decoy, and irrelevant facts is crossed while their text is held fixed (480 tasks). Across Qwen2.5-7B, Llama-3.1-8B, and two DeepSeek models, both heuristics fail in predictable places. Selecting by relevance readmits decoys, because decoys are on topic: even a perfect relevance filter loses 23 to 31 points to rarest-first selection in the classic hidden-profile configuration. Rarest-first selection is no better than random selection on average and wins only where the rare facts happen to be the useful ones. In both cases accuracy follows the balance of evidence the memory keeps, not the policy that kept it. The decision model adds a third failure, a form of social proof: it gives extra weight to facts attributed to many agents. Removing the attribution tags, or reassigning them at random with line order fixed, raises accuracy in the classic configuration by 13 to 24 points. In natural discussions the evidence is lost before any memory acts: the complete transcript yields lower accuracy than no notes at all.