Understanding Positional Embeddings in Modern LLMs
TL;DR Highlight
A walkthrough of how LLMs encode token position, from GPT-2 to RoPE
Who Should Read
A backend developer who works with LLM APIs or model serving and has skipped over RoPE and context-length settings
Core Mechanics
- Without position info, word order is lost
- "The red fox jumps over the gray fox." and "The gray fox jumps over the red fox." differ in meaning only by word order.
- The same token ID (e.g. fox=2) gets the same embedding regardless of position.
- So most LLMs add positional information explicitly.
- GPT-2 learns a table of position vectors
- It adds a position vector to the token embedding before the first transformer block.
- The original GPT-2 has 1,024 position vectors, so it handles only inputs up to 1024 tokens.
- The context length must be fixed before training starts.
- Longer context makes the position table huge
- In GPT-2 Medium, position embeddings are 0.30% of weights at 1,024 positions.
- At a now-common 100k context, they would exceed 22% of total weights.
- The original transformer used fixed sin/cos
- The 2017 "Attention Is All You Need" model computes position vectors with sine and cosine and does not train them.
- They can be evaluated beyond training positions, but good performance isn't guaranteed.
- The authors hypothesized it makes relative offsets like "three tokens earlier" easier to learn.
- RoPE rotates query/key inside attention
- RoPE (Rotary Position Embedding) is a relative method from the 2021 RoFormer paper.
- It rotates query and key by token position before the dot product, and values are not rotated.
- Architectures like MiniMax-M3 add no separate position vector to the input embedding.
- The causal mask hints at order too
- GPT-style causal masks give implicit order info by limiting each token to itself and earlier tokens.
- But they can't tell a separation of one position from ten, so explicit positions still help.
- The article says it covers NoPE, which omits explicit positions, later.
Community Reactions
How to Apply
- Check which position method your model uses
- Find the position encoding (learned absolute, sin/cos, RoPE) in the config and architecture docs of the open-weight model you use today.
- Write one line in your team doc on how it links to the context length limit.
- Check how context can be extended before you extend it
- For learned absolute positions, the table size is the context cap, so truncate inputs to it.
- For sin/cos, computing past the training length works but performance isn't guaranteed, so run a separate long-input eval.
- Locate where RoPE is applied in model code
- Confirm RoPE is applied to query and key inside attention and that value is excluded.
Try It
# Paste into Claude or ChatGPT as-is (simplified version of the article's idea: checking a position encoding method)
You are an LLM architecture explainer. Read the model config or code below and answer.
{{model config or model code}}
Questions:
1. Which method inserts position info (learned absolute, fixed sin/cos, RoPE, other)?
2. Is it added to the input embedding or applied to query/key inside attention?
3. What constraint does this impose when extending context length?
If the code gives no evidence, answer 'cannot confirm'.
Next step: ask the same questions about other model configs and compare the results in a table.Terminology
Related Papers
Gemini 4 Neon
Gemini 4 Neon cuts cost per task 32% versus Carbon, at $3 input and $15 output per million tokens
AI is killing All About Berlin
After Google AI Overviews, a Berlin guide site lost 70-75% of its traffic.
Not Everything Needs an LLM
PostHog used small decision models to skip 98.7% of expensive LLM calls
GPT‑6.2 Astra and Sol
GPT-6.2 Astra and Sol cut costs sharply, but over-engineering and early-stop reports are mixed
Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium
One trn2.48xlarge serves GLM-5.3-Flash through an OpenAI-compatible API using NKI megakernels.
Know When to Zoom: How Small Is Too Small for Vision-Language Models?
Splitting large photos into tiles raised Claude and Gemini accuracy by 11.1–14.5 points
Show HN: DeepSeek V3 in ~200 lines of pure Jax
DeepSeek V3's core design in 190 lines of JAX, trained on a MacBook Air in 2.6 hours
Nvidia allegedly engineered scarcity by funding AI startups to buy its GPUs
WSJ alleges NVIDIA funded startups that then bought its own GPUs to prop up demand.