LLMs Learn Better In-Context from Rules than from Examples | AI Paper Digest

TL;DR Highlight

Describing rules in the prompt beat few-shot examples by 17 accuracy points on average.

Who Should Read

An LLM app developer who stuffs long few-shot examples into prompts, gets inconsistent results, and wants to restructure them.

Core Mechanics

Rules beat examples
- Open-weight models (7B–32B) were more accurate with rules only than with examples only.
- Five tasks across games, arithmetic and linguistic inference used the same test items.
Adding examples to rules doesn't help
- Rules plus examples was not significantly better than rules alone.
- The pattern held across all tasks.
More examples don't reliably help
- Many task/model combinations showed flat or negative gains versus rules as examples scaled up.
- Instruct models fell into long repetitive pattern analysis with many examples and got truncated.
Rules help some tasks more
- Gains were largest for Operator Function (sequential operations) and Set Game (same/different judgments).
- Gains were small or not significant for Lexical Category Inference, which relies on parametric knowledge.
Instruction tuning boosts rules only

Evidence

Rules vs examples +0.171
- In the mixed-effects model (LMM), rules beat examples by 0.171 accuracy (p<.001).
- Per task: Operator Function +0.401, Set Game +0.319, Noun Class Agreement +0.208.
Examples on top of rules: +0.012
- Combined beat rules-only by just 0.012, not significant.
- Per-task differences ranged from -0.044 to +0.064.
Instruction tuning effect
- Instruct vs base: +0.213 with rules and +0.178 with combined (both p<.001).
- With examples it was -0.024, not significant.
Difficulty and model size
- Versus easy, medium dropped -0.073 and hard -0.150, with nearly parallel slopes across the three conditions.
- Larger models (14–32B) beat smaller ones (7–12B) by +0.063 to +0.064.

How to Apply

Rewrite a few-shot prompt as rules today
- Pick one example-heavy prompt and write the input format, output format and decision criteria as numbered rules.
- Compare accuracy against the old prompt on the same 20–50 test cases.
Spell out output format and edge cases
- Add one line each for label set, output format and exceptions you used to convey by example.
- The paper's rule prompts also stated the allowed answer format.
Sort tasks by whether rules help
- For calculation, logical conditions and step-by-step procedures, start with rules.
- For tasks needing semantic knowledge or stylistic feel, rule gains may be small, so A/B test.
Measure before adding more examples
- Run rules-only and rules+examples on the same eval set and drop the examples if no difference.
- You also save tokens.

Try It

copy & run

# Paste into Claude/ChatGPT (simplified version of the paper's method: rules vs examples)

Process the same input {{test input}} with each of the two prompts below and compare the results in a table.

[Prompt A: rules only]
Task: {{task description}}
Rules:
1. {{rule 1}}
2. {{rule 2}}
3. {{exception/edge case}}
Output format: {{allowed answer format}}
Input: {{test input}}

[Prompt B: examples only]
{{input1}} → {{output1}}
{{input2}} → {{output2}}
{{input3}} → {{output3}}
{{test input}} →

Next to try: also run a version C that adds B's examples to A, and see whether it differs from A.

Terminology

In-context learningA model's ability to do a new task from descriptions or examples in the prompt without any weight updates. It's like reading a note and following it on the spot.

Instruction tuningExtra training that teaches a model to follow instructions. Like teaching a new hire how to read the operations manual.

Base modelA pretrained model that hasn't gone through instruction-following training.

Minimum coverage examplesA minimal set of examples picked so every output label and key primitive appears at least once.

LMM (linear mixed-effects model)A statistical method that estimates accuracy differences between conditions while accounting for group-level variation, like model family.

Related Papers

Related Resources

Original Abstract (Expand)

Large language models (LLMs) exhibit in-context learning capabilities, where they can learn new tasks from prompt contexts without weight updates. We compare the learning efficacies of two prominent modes of in-context learning: (1) learning from descriptions of rules (instruction following); and (2) learning from examples of input-output demonstrations (few-shot prompting). Through five learning tasks that cover diverse domains (games, arithmetic, linguistic inferences), we compare two modes of learning (rules vs. examples) specifying the same underlying task. We furthermore explore model and task properties that modulate the learning efficacies. We find that models generally learn more reliably from rules than from examples alone, and additional examples on top of rules or simply scaling up the number of examples do not lead to consistent and significant gains. Instruction tuning amplifies the benefit of rule-based learning while keeping example-based learning capacities intact. Surprisingly, we find no privileged effect of example-based learning in base models, and rules still lead to gains in algebraic task domains. Overall, the comparative efficacy of rules over examples is larger when the task recruits algebraic abstractions and computations, and smaller when the task requires distributional sensitivity and/or recruits parametric knowledge.