6 Hours of YouTube and 85 handtyped pages of notes on LLM Inference – Start here
TL;DR Highlight
Build a Llama 3.2 1B inference engine from scratch in C++/CUDA, with videos and notes
Who Should Read
Backend developer who only uses LLM APIs and wants to understand serving internals like KV cache and batching
Core Mechanics
- Llama 3.2 1B built by hand in C++/CUDA
- It follows a prompt through tokens to the next token on the GPU.
- Weight loading, embeddings, RMSNorm, RoPE, attention and the SwiGLU FFN are built one by one.
- Prefill and decode are separate phases
- Prefill processes the prompt in one pass.
- Decode generates the reply one token at a time, and the KV cache avoids repeated computation.
- Two batching modes: static and continuous
- Static waits for the current group to finish before admitting the next.
- Continuous admits queued requests as soon as page budget frees up.
- Paged KV cache manages memory
- A physical page holds 16 tokens of K and V, with a block table per sequence and layer.
- Pages are allocated lazily.
- Conservative admission policy
- Each request's maximum processed length is reserved up front, and others wait in FIFO order.
- No preemption or recomputation, but a large output budget can limit concurrency.
- Output can differ with batch size
- BF16 rounding can vary across batch sizes.
- With near-tied logits, greedy output can differ between single and batched runs.
Community Reactions
How to Apply
- Watch Parts 0 and 8 to get the vocabulary
- In 30 minutes today, watch Part 0 (AI inference explained) and Part 8 (Decode and KV cache).
- Note where prefill, decode and KV cache show up in your service's latency.
- Use --stats to see batching in action
- With a CUDA GPU, build the repo and run --static and default continuous mode on the same prompts.
- Compare peak active requests, admission waits and final free pages.
- Probe concurrency limits via KV pool size
- Lower --kv-cache-mib, for example to 8, and see when requests start waiting.
- Raise max_new_tokens and check whether concurrency drops.
- Add batch-vs-single output comparison tests
- In serving tests, compare single and batched outputs without requiring exact equality.
- If tokens differ, first check whether the logits were nearly tied.
Try It
# Paste into Claude/ChatGPT (a simplified study version based on the repo's structure)
I'm a backend developer who uses LLM APIs. Explain the concepts below from my service's point of view.
- prefill vs decode
- KV cache
- static batching vs continuous batching
- paged KV cache (16 tokens per page)
My service: {{service description}}
Average prompt length: {{prompt tokens}}, average output length: {{output tokens}}
For each concept give (1) a one-line analogy (2) its effect on my service's latency and cost (3) one metric to check.
Follow-up: explain, using the admission-budget idea, why a large max_new_tokens can reduce concurrency.Terminology
Related Papers
Gemini 4 Neon
Gemini 4 Neon cuts cost per task 32% versus Carbon, at $3 input and $15 output per million tokens
AI is killing All About Berlin
After Google AI Overviews, a Berlin guide site lost 70-75% of its traffic.
Not Everything Needs an LLM
PostHog used small decision models to skip 98.7% of expensive LLM calls
GPT‑6.2 Astra and Sol
GPT-6.2 Astra and Sol cut costs sharply, but over-engineering and early-stop reports are mixed
Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium
One trn2.48xlarge serves GLM-5.3-Flash through an OpenAI-compatible API using NKI megakernels.
Know When to Zoom: How Small Is Too Small for Vision-Language Models?
Splitting large photos into tiles raised Claude and Gemini accuracy by 11.1–14.5 points
Understanding Positional Embeddings in Modern LLMs
A walkthrough of how LLMs encode token position, from GPT-2 to RoPE
Show HN: DeepSeek V3 in ~200 lines of pure Jax
DeepSeek V3's core design in 190 lines of JAX, trained on a MacBook Air in 2.6 hours