Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium
TL;DR Highlight
One trn2.48xlarge serves GLM-5.3-Flash through an OpenAI-compatible API using NKI megakernels.
Who Should Read
An infrastructure or backend developer weighing AWS Trainium instead of GPUs for serving open models.
Core Mechanics
- Open inference engine for Trainium2
- TrnRT is an open-source LLM inference engine for AWS Trainium.
- It serves an OpenAI-compatible API from one trn2.48xlarge.
- Decode fused into one kernel per layer half
- Decode runs as NKI megakernels (many ops fused into one kernel).
- Roughly one kernel per layer half.
- Only GLM-5.3-Flash is supported
- GLM-5.3-Flash is a 320B MoE with 18B active, FP8, 1M-token context.
- Other open models are listed as coming soon.
- Prefix cache spans device and host
- A radix tree sits over FP8 KV pages.
- Evicted pages and state checkpoints go to a host-memory tier.
- Compile ahead of time on a CPU host
- All graphs build on an x86_64 host with no NeuronCore, taking hours.
- The serving box loads them and compiles nothing by default.
- Decode is not bandwidth-bound, they say
- Small instructions, kernel boundaries and exposed collectives dominate decode.
- So kernels keep the working set in SBUF, use both physical cores symmetrically, and move collectives into the kernel.
Community Reactions
- Tokens per minute seen as an odd metric
- One commenter said tok/s is the usual unit and per-second numbers would ease comparison with alternatives.
- Trainium ecosystem called weak
- A commenter was surprised the ecosystem is so poor given Amazon's size.
- They also saw making Trainium a seamless Nvidia substitute as a flywheel play.
- Questions about cost vs Nvidia and AMD
- A commenter asked how token/$ on Trainium compares with Nvidia and AMD chips.
- Question about SGLang
- Another commenter asked how this compares with SGLang.
How to Apply
- Point your existing OpenAI client at it
- If you have a server, change base_url to http://localhost:8000/v1.
- Fix session_id in extra_body for the same conversation to hit the prefix cache.
- Convert the benchmark into your cost table
- Put the README's $/M tokens ($0.040 at batch 12 on $8.60/h spot) next to your current GPU serving cost.
- Include the $0.167 figure at $35.76/h capacity block too.
- Pick a batch setting for agent workloads
- Consider batch 4 or 8 when latency matters and the default batch 12 for throughput.
- Note that batch 12 has a 6.53 s cached first token (p50).
Try It
# Python client for a running TrnRT server with an OpenAI-compatible API (pip install openai; start the server per README step 3)
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
reply = client.chat.completions.create(
model="glm",
messages=[{"role": "user", "content": "Write a haiku about Trainium."}],
extra_body={"session_id": "my-session"}, # keeps a conversation on one engine's prefix cache
)
print(reply.choices[0].message.content)
# Next: call repeatedly with the same session_id on a long conversation and compare latencyTerminology
Related Papers
Gemini 4 Neon
Gemini 4 Neon cuts cost per task 32% versus Carbon, at $3 input and $15 output per million tokens
AI is killing All About Berlin
After Google AI Overviews, a Berlin guide site lost 70-75% of its traffic.
Not Everything Needs an LLM
PostHog used small decision models to skip 98.7% of expensive LLM calls
GPT‑6.2 Astra and Sol
GPT-6.2 Astra and Sol cut costs sharply, but over-engineering and early-stop reports are mixed
Know When to Zoom: How Small Is Too Small for Vision-Language Models?
Splitting large photos into tiles raised Claude and Gemini accuracy by 11.1–14.5 points
Understanding Positional Embeddings in Modern LLMs
A walkthrough of how LLMs encode token position, from GPT-2 to RoPE
Show HN: DeepSeek V3 in ~200 lines of pure Jax
DeepSeek V3's core design in 190 lines of JAX, trained on a MacBook Air in 2.6 hours
Nvidia allegedly engineered scarcity by funding AI startups to buy its GPUs
WSJ alleges NVIDIA funded startups that then bought its own GPUs to prop up demand.