Not Everything Needs an LLM
TL;DR Highlight
PostHog used small decision models to skip 98.7% of expensive LLM calls
Who Should Read
A backend developer sending even simple yes/no checks to a large LLM and struggling with cost and latency
Core Mechanics
- Decision models return a probability with the answer
- You ask a structured question and get a typed answer such as 'Yes, 0.99'.
- In a calibrated model, answers scoring 0.8 are right about 80% of the time, and 0.5 is a coin flip.
- Use them as gates before expensive LLM calls
- The article says they are up to 444.6x cheaper and 193.6x faster than frontier models, citing TypeSafe AI.
- In Replay Vision's PII scrubbing, each summary gets 2 yes/no questions, and a score of 0.4 or above sends it to Gemini for rewriting.
- In production, 98.7% skip the LLM call entirely.
- Set thresholds by cost of mistakes
- A 0.85 score still means about 150 wrong answers per 1,000 checks.
- For PII, a false negative is worse, so the threshold is 0.4.
- The Slack bot's false positives are annoying, so it only lets 0.85+ through.
- They also work as model routers
- Each option gets a short note, and the model picks the cheapest one likely to finish on the first try.
- Thresholds differ by task, model and version, and changed even between v1 and v2 in Error Tracking.
- Pre-process input so one pass is enough
- Embeddings pick candidates first, then the model only checks how well each matches the search term.
- This improved ranking quality by about 25% on 20 test queries.
- Multi-step reasoning failed, as an hourly job with up to 32 sessions of 12-38k tokens each blocked the model server for hours.
- Open-ended tasks still belong to LLMs
- Ranking PR review agent findings gave an AUC of 0.52, which was inconclusive.
- Jev takes 64k tokens per request and 32k per question by default, and its long-context mode is slower and less accurate.
- Jeeves, a hybrid that reasons when unsure, is PostHog's default.
Community Reactions
- Opinion that model routing is risky
- A commenter said routed models may be newer than the decision model, which then knows only a short blurb about them.
- Pushback on trusting the probability
- Commenters asked how we know 0.85 isn't really 0.75 or 0.9.
- The point is that calibration needs evidence.
- Opinion that LLMs are safer without evals
- One commenter said that unless you can run online and offline evals, use the most expensive LLM.
- They had seen bad evals that missed real-world distributions and broke products.
- Skepticism that it differs from structured output
- One commenter has used structured JSON outputs since GPT-4 and didn't see the hype.
- Another asked whether a decision model is just an LLM told to limit its responses.
- Concern about customer data and PII
- One commenter said PostHog shouldn't use these on customer data.
- Another found PII protection a questionable example.
How to Apply
- Add a gate question before one costly LLM call today
- Pick a call in your logs that mostly gets passed or ignored.
- Put one or two yes/no questions in front to check whether it is needed.
- Keep the questions narrow and measure the pass rate first.
- Set thresholds by cost of errors
- Write down whether false positives or false negatives cost more.
- Lower the threshold when misses are dangerous, and raise it when false alarms are annoying.
- Recalibrate with hand-labeled samples whenever you change models or versions.
- Split complex judgments into one question after preprocessing
- For ranking, narrow candidates with embedding search first.
- Ask the decision model only whether each candidate matches.
- Leave multi-step reasoning to the LLM.
- Make thresholds configurable and run in shadow mode
- Put thresholds in per-project or per-task settings.
- When swapping models, run in shadow mode that only logs decisions and compare.
Try It
# Paste into Claude or ChatGPT (a simplified prompt-level imitation of the article's decision model approach)
You act like a decision model. Output JSON only, with no explanation.
Format: {"answer": "yes" or "no", "probability": 0 to 1}
Question 1: Does the text below name a real person?
Question 2: Does the text below show contact details or an ID?
Text: {{document}}
Output one JSON line per question.
Next step: run 20 sample documents, compare against your own labels, and count misses and false flags at thresholds 0.4 and 0.85.Terminology
Related Papers
Gemini 4 Neon
Gemini 4 Neon cuts cost per task 32% versus Carbon, at $3 input and $15 output per million tokens
AI is killing All About Berlin
After Google AI Overviews, a Berlin guide site lost 70-75% of its traffic.
GPT‑6.2 Astra and Sol
GPT-6.2 Astra and Sol cut costs sharply, but over-engineering and early-stop reports are mixed
Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium
One trn2.48xlarge serves GLM-5.3-Flash through an OpenAI-compatible API using NKI megakernels.
Know When to Zoom: How Small Is Too Small for Vision-Language Models?
Splitting large photos into tiles raised Claude and Gemini accuracy by 11.1–14.5 points
Understanding Positional Embeddings in Modern LLMs
A walkthrough of how LLMs encode token position, from GPT-2 to RoPE
Show HN: DeepSeek V3 in ~200 lines of pure Jax
DeepSeek V3's core design in 190 lines of JAX, trained on a MacBook Air in 2.6 hours
Nvidia allegedly engineered scarcity by funding AI startups to buy its GPUs
WSJ alleges NVIDIA funded startups that then bought its own GPUs to prop up demand.