The Instruction Tax on Factual Recall: Base Models Can Beat Their Instruction-Tuned Counterparts
TL;DR Highlight
Few-shot completion with repeated sampling on a base model beat chat: F1 0.684 vs 0.568
Who Should Read
A backend developer building a factual extraction pipeline who is struggling with hallucinated or inaccurate LLM answers
Core Mechanics
- Instruction tuning erodes factual recall
- With the same gemma-3-27b weights, the base model scores 0.684 and the it model via chat scores 0.568.
- The authors call this loss the instruction tax.
- They attribute it to the weights, not the prompting method.
- Don't ask questions, make it continue text
- Show a few examples and leave the last line unfinished as a completion prompt.
- Use no chat template and no instructions.
- The it model still scores lower on the same completion prompt.
- Sample 64–256 times and keep what repeats
- Sample many outputs per query and keep answers whose frequency passes a threshold.
- Return an empty answer if nothing passes.
- Fit the threshold per relation on training data.
- The tax size varies by tuning run
- gemma-3 models pay a large tax, while gemma-2-27b and Mistral-Nemo pay a small one.
- With a high tax, correct answers vanish from the samples entirely.
- With a low tax, answers still appear but selection fails.
- A one-GPU-hour probe picks the model
- Use about 100 known-answer examples with several phrasings, sampled 8–12 times.
- The probe ranking matched the official submission ranking.
- Model version and interface can be chosen before building the full system.
- Knowing when not to answer boosts the score
- Before asking for place of death, check whether the person is alive using 16 samples.
- If all 16 say alive, output the empty set.
- Rare candidates get a second yes/no verification pass.
Evidence
- Base completion beat it chat
- Official test F1 is 0.684 for gemma-3-27b-pt completion vs 0.568 for gemma-3-27b-it chat (+0.116).
- The leaderboard score is 0.6836 vs the task baseline of 0.2964.
- The hasArea gap was the biggest
- Official test F1 is 0.820 for base vs 0.617 for it chat (+0.203).
- hasCapacity is 0.316 vs 0.160 (+0.156).
- Correct answers vanished from the samples in the probe
- On the hasArea probe, gemma-3-27b ceiling dropped from 0.95 to 0.69.
- gemma-3-12b dropped 0.84→0.47, and gemma-2-27b 0.82→0.77.
- More samples and extra steps raised the score
- Official score was 0.6218 at n=8, 0.6292 at n=24, and 0.6604 at n=64.
- Verification added +0.027 on awardWonBy and the existence check added +0.030 on the death relation.
How to Apply
- Compare completion vs chat on your current model
- Pick one factual extraction prompt and run a chat-question version and a few-shot continuation version on the same model.
- Compare accuracy on 100 samples with known answers.
- Include a base model if you can.
- Sample many times and keep the consensus
- Sample each query about 64 times at temperature 0.7.
- Count how often each answer appears and return only those above a threshold.
- Fit the threshold on labeled data and return empty if nothing passes.
- Run an existence check before detail questions
- For fields often empty, such as death info, ask whether a value exists first.
- Ask for the detail only when confident it exists.
- Use exemplars split half and half between exists and not.
- Rescue low-frequency candidates with yes/no verification
- For list-type questions, collect rarely seen candidates separately.
- Sample a yes/no completion with 5 positive and 5 negative examples eight times.
- Add back candidates with a yes ratio of at least 0.75.
Try It
# Paste into a base model or completion API (simplified version of the paper's method, no chat template)
The total area of Lake Victoria is 68800 square kilometres.
The total area of Bodensee is 536 square kilometres.
The total area of {{subject name}} is
# Sample this prompt 64 times at temperature 0.7, cut at the first line break, then take the median of the largest cluster within ±5% as the answer.
# Next to try: ask the same question directly to a chat model and compare accuracy.Terminology
Related Papers
AI is a tool and using it well takes skill
A rebuttal to Hickey: LLMs are tools, and skill in directing and checking them decides the outcome
Reflective Prompt Optimization of a Deep-Research Agent: A Topic-Disjoint, Multi-Seed Protocol for Measuring Transferable Gains and Prompt-Level Overfitting
Reflective prompt optimization raised held-out citation reward from 0.702 to 0.781 on a deep-research agent.
GPT‑6 and Intelligent UI for everyone
GPT-6 reportedly adds generated UI to chat; with no article text, only comment reactions are available.
Base Models Can Reason By Taking a Cue From Training Data
Prefilling just two opening tokens lifted Olmo-3-7B's MATH-500 accuracy from 42% to 78%.
Labels Override Definitions in Jev-Style Typed Decision Models
Renaming options to A/B raised routing accuracy by +0.1511 over meaningful labels.
How Much Prompt Is Enough? A Blackbox Minimization of Few-Shots in LLMs
Delta debugging trimmed few-shot prompts by 65.3% on average with unchanged propositional output
Language Models Are "Insecure" Reporters
Adding "Be honest" raised GPT-5.5's flagging of a hidden negative result from 2/200 to 190/200.
Prompting Claude Opus 5.5
Opus 5.5 at medium effort matches Opus 5 at high; guide covers prompt and harness tweaks
Related Resources
Original Abstract (Expand)
We present our solution to the AKBC LM-KBC 2026 Shared Task, where we achieved a score of 0.6836 on the official leaderboard, more than double the baseline of 0.2964. This task requires generating all possible objects for subject-relation pairs using a language model with up to 32 billion parameters, with no retrieval or fine-tuning involved. We show that the prompting methodology is as important as the model architecture: while the instruction-tuned model version achieves 0.5681 with its chat interface, the base model achieves 0.6836 when prompted using text completion. Thus, instruction tuning decreases factual recall; we refer to this effect as the instruction tax, which is caused by the model's weights rather than the prompting method. Our final solution is fairly straightforward: we prompt the base model with some examples from the training data, generate 64--256 outputs, and retain only the repeated ones. Two specialized methods are then applied for each relation: one improves the recall of large answer sets, and the other verifies a person's vitality before predicting their place of death.