Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments? | AI Paper Digest

TL;DR Highlight

On 20 unfamiliar-rule games, keeping raw history (MEMORY) beat more elaborate summarizing self-evolving methods.

Who Should Read

A backend developer deciding whether to give an agent raw past run logs or summarized memory.

Core Mechanics

Raw history beat summaries
- With Kimi K3, MEMORY (accumulating unmodified earlier episode histories in the prompt) led all four metrics.
- The paper says summarizing can harden an incomplete understanding into rules that later episodes keep following.
One failure became a ban
- In GemForge, EvoTest-Opus 5 added a rule to avoid many other combinations after one failed combination.
- Those included a combination that would have succeeded.
Same model, different harness, different results
- With the backbone fixed, switching the harness from OpenCode to Claude Code or Codex raised all four metrics.
- Some configurations also cost less.
Top scorer is not top learner
- Opus 5.5 led Max and Mean, while Qwen 3.8 Max led LG and LS.
- The highest-scoring model is not necessarily the one that improves most.
Knowing rules is not planning

Evidence

MEMORY beat summarizing methods with Kimi K3
- With Kimi K3, MEMORY scored Max 63.3 and LS +4.69, versus Reflexion 61.6/+3.35, ReasoningBank 59.9/+2.59, and AWM 56.1/+0.62.
- NAIVE (no cross-episode memory) scored Max 54.7 and LS +0.42.
Changing only the harness raised scores
- Opus 5 went from Max 74.5 on OpenCode to 81.8 on Claude Code.
- GPT-5.6-SOL went from 57.0 on OpenCode to 74.3 on Codex.
Human Top-1 versus agents
- Max was 84.3 for Human Top-1, 81.8 for Claude Code+Opus 5, and 80.1 for OpenCode+Opus 5.5.
- Consecutive-episode action similarity was 0.54 for humans and 0.66 for agents.
- After a score drop, a new personal best came next 33% of the time for humans and 22% for agents.
The harness also cut cost
- With Kimi K3, OpenCode used 56% fewer input tokens than MEMORY.
- It generated 71% fewer tokens than EvoTest and 41% fewer than ReasoningBank.
- Cost per episode was $0.313 for OpenCode, $0.586 for MEMORY, and $0.658 for EvoTest.

How to Apply

Compare your agent memory with raw logs
- If you use summarized memory, run the same task with the full raw history of earlier attempts and compare scores.
- If context is tight, drop the oldest episodes first, as the paper's MEMORY method does.
Link each summarized rule to its evidence
- Store the raw-log location behind every rule extracted by reflection or summarization.
- When new feedback conflicts with a rule, have the agent reread the original and revise it (a direction the paper suggests).
Check attempt counts before writing bans
- Prevent summary prompts from creating "never do X" rules from a single failure.
- Require the number of attempts and results alongside each rule.
Compare harnesses before swapping models
- Run the same model under different harnesses (e.g. Claude Code, Codex, OpenCode) and measure score and token cost together.
- Setups that batch several actions in one response and trim repeated feedback cut cost.

Try It

copy & run

# Paste into Claude or ChatGPT — simplified version of the paper's method (keep raw history across repeated attempts)

You are an agent learning a task with hidden rules over repeated attempts.
Below are the raw records of earlier attempts. Do not summarize them; use them directly as evidence.

[Earlier attempt logs]
{{full logs of earlier attempts}}

[Current task]
{{task description}}

Instructions:
1. Separate confirmed facts from guesses. Attach the log location that supports each fact.
2. Mark anything tried only once as "unverified", not as a ban.
3. Propose the next attempt, changing only one factor at a time.
4. Even if a strategy already works, decide whether to test one new strategy and explain why.

Try next: run the same task with only a summary instead of the logs and compare results.

Terminology

HarnessThe wrapper that decides tool calls, context handling, and step order around a model. It is like fitting different bodywork on the same engine.

Self-evolving agentAn agent that keeps model weights fixed and improves by reusing past experience through memory or prompts.

Learning Gain (LG)The difference between average scores in the last few and first few episodes of evaluation.

Learning Slope (LS)The slope of a line fitted to per-episode scores, showing improvement per episode.

Re-shuffled gameA game whose rules stay the same while surface details like characters or menus change. It tests whether agents generalize rules.

Related Papers

Related Resources

https://liushiliushi.github.io/learn2play-bench-website/

Original Abstract (Expand)

Learning from experience is essential for LLM agents to adapt to unfamiliar and dynmaic environments. Evaluating this ability is therefore important for understanding how effectively agents acquire and use new knowledge. Existing benchmarks have sought to evaluate this ability, but they primarily evaluate tasks whose rules are provided in the instructions or already familiar to pretrained models, making it difficult to distinguish learning from interactions from reasoning with existing knowledge. To address this, we introduce Learn2Play Bench, a benchmark of newly designed text-based games, whose rules are novel or counterintuitive, requiring agents to acquire knowledge through interaction rather than rely solely on pretrained knowledge. These games provide reproducible feedback and automatic scoring, enabling controlled evaluation of learning across repeated attempts. We also vary game instances to test whether agents can apply what they have learned to new situations. Therefore, we evaluate how backbone models, self-evolving methods, and agent harnesses affect agents' learning ability, revealing three findings: (1) Experience retention: Retaining complete records of actions and feedback can support more effective learning than summarizing these experiences into rules or strategies. (2) Human agent gap: Top-performing human players achieve higher peak scores than the evaluated agents. Human explore more varied strategies, and repeat actions less. (3) Harness matters: With the backbone fixed, changing the harness can improve performance while reducing estimated inference cost. Together, these findings provide insights into how LLM agents learn from experience and suggest directions for future work to improve their learning ability. Project website: https://liushiliushi.github.io/learn2play-bench-website/