If you don't run your own LLM proxy, how do you know whats going on?

TL;DR Highlight

Running your own LLM proxy shows which keys, models, and cache misses drive your usage.

Who Should Read

A developer using agents like Claude Code or Codex who can't see token usage or why cache misses happen

Core Mechanics

Vendor analytics panels aren't useful
- The author says provider Analytics panels aren't time-series token stats meant for engineers.
- He says he can't make decisions from them.
One proxy unifies providers
- You get multiple providers and models behind a single endpoint you control.
- Switching models or providers doesn't mean redoing setup from scratch.
Logs show which key called what
- Logs record which key called which model, from which client, when, and with what status.
Virtual keys protect your main key
- You can give each job, agent, and harness its own virtual key.
- Your main key isn't scattered everywhere.
Cache misses diagnosed in 20 seconds
- The author says he can pinpoint a cache miss in a coding session by hand in 20 seconds.
- He hasn't automated it, but his proxy has an API that makes it possible.

Community Reactions

OTEL may be enough without a proxy
- One commenter says Codex and Claude Code emit OTEL traces covering tool calls, prompts, tokens, and cost.
- They argue a proxy is a single point of failure that can strand the team.
Subscription users can read session files
- One commenter says collecting JSONL files from ~/.claude/projects/ is better organized, e.g. subagents.
- They say proxies suit API pricing.
API cost is a concern
- One commenter on Claude Max 20x ($200/month) says API keys hit $200 in 2-3 days.
LiteLLM concerns and alternatives
- One commenter is setting up a LiteLLM proxy to issue virtual keys to sandboxed agents.
- Others worry about its bugs and vulnerabilities, and one is happy using bifrost instead.

How to Apply

Turn on Claude Code OTEL output first
- Read the Claude Code monitoring docs today and enable OTEL export.
- Check where token usage and tool calls show up.
Issue a virtual key per agent
- When adopting a proxy, give each agent or sandbox its own virtual key.
- Keep the provider's original key out of sandboxes.
Define log fields for cache-miss analysis
- Log key, model, client, time, and status for each request.
- Compare a missed request against its neighbors to find the cause.
Pick observability by pricing plan
- If you pay API rates, set up a proxy.
- If you're on a subscription, try session JSONL or OTEL first.

Try It

copy & run

# Prompt to paste into Claude Code (simplified version of the article's idea)

Help me design observability for my LLM calls. Use the info below.
- Tool: {{Claude Code / Codex / other}}
- Plan: {{API keys / subscription}}
- What I want to see: {{cache-miss cause / per-key usage / per-model cost}}

Requests:
1. Pick proxy, OTEL, or session logs for this situation and explain why.
2. Make a table of log fields I must keep (key, model, client, time, status, token counts).
3. Give me a checklist to follow when a cache miss happens.

Next step: paste a few real log lines and ask it to analyze the cache miss.

Terminology

LLM proxyA middle server between your app and model providers. Every call passes through it, so logging and key management happen in one place.

Virtual API keyA stand-in key issued by the proxy. The master key stays in the vault while each person gets their own badge.

OTel (OpenTelemetry)A standard format for exporting logs and traces. It's a common report form apps use to describe their own state.

HarnessThe wrapper that runs a model, handling tool calls and conversation. Tools like Claude Code are examples.

Cache missWhen a previous prompt cache can't be reused and must be recomputed. Cost and latency go up.