A simple fix for LLM tail latency

TL;DR Highlight

Sending each LLM request twice cut worst-case latency from 9.8s to 3.5s versus priority tier.

Who Should Read

A backend developer whose realtime voice agent suffers occasional 10-second LLM stalls

Core Mechanics

Rare 10-20s responses are the problem
- Most responses return within 1.5 seconds, but a few take 10 to 20 seconds.
- On a phone call, that silence makes callers hang up.
More turns mean higher odds of a stall
- A typical call has 20 to 30 turns.
- If 1% of requests are very slow, a 25-turn call has about a 22% chance of a long silence.
Fix: send every request twice, use the faster
- They stayed on the standard tier and took whichever response came first.
- The comparison was OpenAI's priority tier at 2x cost per token.
Time to first token beat priority tier
- They replayed 50 real production requests.
- Median 0.61s→0.58s, p95 1.04s→0.68s, p99 4.2s→1.2s (priority vs. sent twice).
Full-response tail dropped sharply
- Median stayed 1.35s, p95 went 3.4s→2.0s, p99 9.8s→3.5s.

Community Reactions

Hedged requests to cut cost
- One commenter said to only send a backup after a threshold such as p95.
- They noted Google calls this a hedged request and linked The Tail at Scale.
Retry after first-token timeout
- Another suggested sending a second request if no first token arrives within about 1 second.
- They noted parallel duplicates double the cost.
Missing cost-performance comparison
- A commenter said the post is half the story without performance per dollar.
Doubts about priority tier value
- One commenter expected more than 2x speed for 2x price.
- Another said self-hosting gives more predictability.

How to Apply

Pull p95 and p99 from your latency logs
- Today, compute p50/p95/p99 for time to first token and full response.
- Flag calls where p99 is 5x the median or more.
Add duplicate sending to realtime paths
- Send the same request twice in parallel on latency-sensitive calls only.
- Use the first completed response and cancel the other.
Compare against priority on real requests
- Replay 50+ real requests on priority tier and on duplicate sending.
- Record latency and token cost together.
Try a threshold-based backup too
- Send the second request only if no first token arrives by your p95 time.
- Compare duplicate cost and p99 gain against full duplication.

Try It

copy & run

# Python 3.9+ (pip install openai) — simple duplicate-send example. Replace the model name and prompt

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI()

async def call(messages):
    r = await client.chat.completions.create(model="{{model}}", messages=messages)
    return r.choices[0].message.content

async def hedged(messages):
    tasks = [asyncio.create_task(call(messages)) for _ in range(2)]
    done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
    for p in pending:
        p.cancel()
    return done.pop().result()

print(asyncio.run(hedged([{"role": "user", "content": "{{question}}"}])))

# Next: run it 100 times, compare p95/p99 against single calls, and log token cost too.

Terminology

tail latencyThe rare very slow responses in an otherwise fast system. Like 1% of restaurant guests waiting an hour.

p95/p99The response time that 95% or 99% of requests beat. It marks the edge of the slow tail.

TTFT (time to first token)Time until the model emits its first token. For a voice agent, when it starts speaking.

priority tierA pricier service level for faster, more consistent responses. Like priority boarding.

hedged requestSending a duplicate request when one is slow and using whichever returns first. Like switching to a faster checkout line.