Skill Blocks: How Should an Agent Load Its Skill? A Caching-Correct Comparison of Pre-load, On-Demand Tool-Loading, Progressive Disclosure, and Hybrid | AI Paper Digest

TL;DR Highlight

Loading agent skills on demand instead of pre-loading cut input tokens up to 73% for large multi-turn skills.

Who Should Read

Agent developers whose long SKILL.md-style skill documents are pasted into every request and driving up token costs

Core Mechanics

There is no universal winner
- Which loading method wins depends on skill size and turn structure.
- The paper concludes that no single method is best everywhere.
On-demand wins big for large multi-turn skills
- On ScienceWorld (~6K-token skill), Skill Block cut effective input by 62.5% and hybrid by 52.8%.
- On SynthProc (~10K-token skill), the cuts were 73.0% and 66.6%.
Tool loading hurts on small skills
- On SearchQA (~2K skill), Skill Block with a forced 'load before answering' instruction raised input by 48.4%.
- Tool schema and extra round-trips outweighed the savings from unused text.
Hybrid survives even small skills
- Short stubs for every block were usually enough, so full fetches were rare.
- On SearchQA hybrid cut input by 27.4% and loaded a block on only 1.4% of queries.
Account for cache reads separately

Evidence

SearchQA (n=1,400), single-turn raw input
- Versus full at 5,535 tokens: hybrid 4,016 (−27.4%), reference 6,112 (+10.4%), Skill Block 8,216 (+48.4%).
- Hard accuracy stayed within .841-.849.
SpreadsheetBench single-turn (276 cases)
- Versus full at 12,461: hybrid 7,506 (−39.8%), Skill Block 8,028 (−35.6%), reference 8,517 (−31.7%).
- original8, which permanently removes six blocks, used 5,547 (−55.5%) but is not a content-parity comparison.
ScienceWorld (n=84), median effective input
- Versus full at 26,810: Skill Block 10,061 (−62.5%), hybrid 12,654 (−52.8%), reference 20,644 (−23.0%).
- Successes were 57/84 full, 62/84 Skill Block, 58/84 hybrid, 50/84 reference, with no significant differences.
SynthProc (n=40) and gpt-5.6 rerun
- SynthProc effective input: full 21,617 vs Skill Block 5,837 (−73.0%) and hybrid 7,221 (−66.6%).
- gpt-5.6 ScienceWorld (n=90): full 28,459 vs hybrid −68.8% and Skill Block −67.8%.

How to Apply

Measure how much of your skill goes unused
- Split the skill into sections and count per-section usage in recent request logs.
- Keep only always-used sections as core and mark the rest as candidates.
Add a load_skill_block tool for big multi-turn skills
- If the skill is thousands of tokens and each turn uses only a few procedures, send just core plus a section list.
- Provide a tool that returns a section's original text by name.
Use hybrid stubs for small skills
- Put a one-line summary of each section in context and load the full section only when the summary is not enough.
- Avoid instructions that force the model to load sections.
Log raw, cache, new, and output tokens separately
- Before comparing multi-turn methods, compute new tokens as input_tokens minus cache_read.
- Vary the cache discount locally and check whether the ranking flips.

Try It

copy & run

# Paste into Claude or ChatGPT (simplified version of the paper's method: a helper for splitting a skill into core/optional)

Below is the skill document for my agent.
{{skill document}}

Analyze it in this order.
1. Split the document into sections (blocks) and tabulate the approximate token count of each.
2. Using the recent request examples below, separate sections needed in every request (core) from those needed occasionally (optional).
{{10 recent request examples}}
3. Write a one-line summary (stub) for each optional section.
4. Draft a new system prompt containing core + the stub list + a description of a load_skill_block(name) tool.

Next steps: compare input token counts of the original and new setups, and check on the same 10 requests that answer quality stays the same.

Terminology

Agent Skill (SKILL.md)A document giving an agent specialized procedures and constraints. It's like pasting a manual into the prompt.

Progressive DisclosureShow only a catalog first, then supply the reference files the model picks. It's like reading a table of contents and opening only the chapter you need.

Cache read tokensInput tokens for the earlier part of the prompt that are served from cache. The paper counts them at 10% in its effective input measure.

Effective inputInput counted with new tokens at 100% and cache reads at 10%. It approximates real multi-turn cost.

McNemar testA statistical test that compares two methods on the same problems by looking only at pairs where correctness differed.

Related Papers

Original Abstract (Expand)

Agent skills are often injected in full on every request, increasing token cost. We compare four content-preserving loading methods: Full, Skill Block, Reference, and Hybrid. Across SearchQA, SpreadsheetBench, ALFWorld, ScienceWorld, and SynthProc, we measure token usage using raw input for single-turn tasks and cache-correct effective input for multi-turn tasks. Results show no universal winner. Hybrid reduces input by 27.4% on SearchQA and 39.8% on SpreadsheetBench. On large multi-turn skills, Skill Block and Hybrid achieve substantial reductions, reaching 62.5% and 52.8% on ScienceWorld and 73.0% and 66.6% on SynthProc. ALFWorld shows smaller gains because procedures are short and repeatedly needed. Paired outcome tests detect no quality differences, though they do not establish equivalence. Overall, conditional loading is most beneficial when large portions of a skill are not needed on every turn.