Token Optimization and Context Window Management in Multi-Agent AI Workflows | AI Paper Digest

TL;DR Highlight

Mixing in low-relevance same-domain items raised relevance-scoring accuracy by 0.077 over all-relevant prompts.

Who Should Read

Backend developers whose LLM pipeline extracts key items from email and meeting notes and who are struggling with token cost and latency

Core Mechanics

Only-important context is worse
- Replacing half of 10 prompt items with low-relevance same-domain items was tested.
- Relevance accuracy (RelAcc) rose +0.077 versus the 100% signal condition, averaged over 11 model configurations.
The gain came mostly from email
- Email target items gained +0.119.
- Meetings (+0.002) and chats (+0.017) were near zero.
Route by source type
- Extracting meetings only raised RelAcc by about +0.16 versus mixed-source prompts (exploratory, not confirmed).
- 73.3% of meeting items were relevant versus 5.1% of emails.
Position matters little at short length
- At 10 items (about 2,000-3,000 tokens), end placement beat start by +0.022.
- It was not significant after Holm correction (p_adj=.098).
Orchestration changes cut latency

Evidence

Relevance accuracy (mean of 11 configs x 20 blocks)
- 100% signal 0.584, 50% 0.661, 30% 0.661, 10% 0.648.
- 50% vs 100% was +0.077 (95% CI [+0.056, +0.098], Cohen's d=0.49, Holm p<.001, n=220).
By model family
- The mean effect over 9 families was +0.084 (95% interval [+0.064, +0.103]).
- All 11 configs were positive, but Claude Opus 4.7 xhigh was only +0.013.
Production pipeline (only 6 of 25 runs timed)
- Cold load went from an estimated 3.5-10.5 minutes to 61-116 seconds.
- Input tokens fell an estimated 60-70%, not exact telemetry.
Corrected-corpus rerun and Fusion-of-N
- The gain was +0.083 on the corrected corpus and +0.092 on the original.
- Fusion was -0.048 versus best-of-N and -0.073 versus union.

How to Apply

A/B test low-signal filler in extraction prompts
- Today, replace half the items in your extraction prompt with irrelevant same-domain items.
- Compare agreement with your gold labels per source type.
Split extraction calls by source type
- Route sources with very different signal density (meetings/email/chat) before the model call.
- Measure each type's signal rate first.
Fetch once, match locally
- Find places where sub-agents re-fetch the same source and switch to one fetch.
- Do not rerun acquisition when retrying with a fallback model.
Try set union before LLM fusion
- For recall-oriented extraction, union the item IDs across samples.
- Add an LLM fuser only if it beats union on your metric.

Try It

copy & run

# Paste into Claude or ChatGPT (simplified version of the paper's method: relevance-contrast context)

Below are 10 workplace communication items. Score each item's relevance from 0 to 3.
3=decision/action/blocker/commitment, 2=status for a weekly summary, 1=passing mention, 0=unrelated to work.
Output only a JSON array [{"id":..., "score":...}].

[5 likely-relevant items]
{{5_relevant_items}}

[5 low-relevance items from the same domain]
{{5_low_relevance_items}}

Next step: rescore the same 5 relevant items in a prompt filled only with relevant items, and compare agreement with human labels.

Terminology

Relevance-contrast contextA prompt mixing important items with less important same-domain ones. It lets the model set a baseline for what counts as unimportant.

RelAccHow closely the model's 0-3 relevance scores match human labels. Exact match gets full credit and off-by-one gets partial.

Semantic cachingReturns a stored pipeline result without calling the model when the request matches. It differs from prefix caching at the model-service layer.

Fusion-of-NAn LLM merges N samples of the same task into one result. The paper compares it with simple set union.

Context stratificationSending each call only the data type it needs. Input per call fell from about 30K to 3K tokens (estimated).

Related Papers

Original Abstract (Expand)

Multi-agent AI workflows are limited not only by model quality but by token cost, latency, and context-window quality. This paper presents a practitioner framework for token optimization and context-window management, grounded in an internal production dashboard that extracts structured work items from meetings, email, and chat with LLMs and routes summaries across workstreams. Six patterns are described: context stratification, fetch-once/process-locally architecture, schema-contracted prompts, token-aware fallback chains, semantic caching, and inter-agent communication compression. In production they cut measured cold-load latency to 61-116 seconds (six timed runs) from an operational baseline of roughly 3.5-10.5 minutes, with an estimated 60-70% token reduction. It also reports a controlled context-composition study: 2,420 confirmatory trials across 11 model configurations, using 661 anonymized workplace items scored for relevance. Holding the prompt at a fixed ten items, replacing some high-relevance items with same-domain low-relevance items improves the model's relevance-score concordance on the target items, versus high-relevance items only; we call this relevance-contrast context. In the all-11 paired analysis, the 50:50 signal/noise condition improved relevance accuracy by +0.077 over the 100% condition (naive 95% CI [+0.056, +0.098], Cohen's d = 0.49, Holm-adjusted p < .001, n = 220). These cells are not independent; by the nine model families the effect is +0.084 (95% interval [+0.064, +0.103]), reported as a within-corpus descriptive comparison, not a population inference. A Fusion-of-N follow-up found that learned synthesis did not beat the mechanical set union of item IDs. The contribution is a measured engineering layer between model research and production agent practice: repeatable patterns and evaluation methods for faster, cheaper, more reliable workflows.