OpenAI Jalapeño: Better than Nvidia Blackwell

TL;DR Highlight

OpenAI's own inference chip Jalapeño beat Rubin's MTP tokens-per-MW using only single-token prediction.

Who Should Read

A backend or infra developer weighing inference chips and cloud providers because of LLM cost and latency.

Core Mechanics

Jalapeño is a general inference chip
- OpenAI and Broadcom designed it from scratch for LLM inference, not just for OpenAI models.
- Design began mid-2024 and reached tape-out in about 16 months.
Tokens per MW beat Rubin's MTP
- With single-token prediction only, Jalapeño's output tokens per MW exceeds Vera Rubin's July MTP results.
- It used no speculative decoding and no prefill-decode disaggregation.
Low-concurrency speed is striking
- On DeepSeek R1 it exceeded 700 tok/s/user at concurrency 1.
- On Kimi K2.5 it hit nearly 700 tok/s/user, over 9x the next chip's 100 tok/s/user.
- GPT-OSS ran at about 1,400 tok/s/user.
TCO is on par with Rubin
- Output tokens per dollar are almost the same as Vera Rubin.
- Rubin used speculative decoding and Jalapeño did not.

Community Reactions

Opinion that it reads like an OpenAI press release
- A commenter noted the piece says they were invited to benchmark, then says OpenAI supplied the numbers.
- Another said it compares an engineering sample to older, widely available chips, a classic PR tactic.
Pushback on speculative decoding optimism
- Commenters said it can speed things up 5x or slow them down 2x depending on workload.
- One suggested it was off by default because it may not suit this ASIC.
Opinion that the chip is compute-bound
- One commenter said big bandwidth paired with weaker cores makes it compute-bound, so MTP gives no gain.
- They guessed the power savings come from a much lower clock.
Expectation that token prices keep falling
- Some said continued hardware gains make falling token prices hard to doubt.
- Another said prices will converge to energy cost, favoring large labs.

How to Apply

Record tok/s/user and cost of your current model
- Measure per-user tok/s on your live endpoint at concurrency 1, 8, and 32 today.
- Generate the script with the prompt below and run it.
Fix three criteria when comparing providers
- Check concurrency, input/output length (e.g. 8k1k), and whether speculative decoding was used.
- Do not compare numbers directly if any of the three differ.
Benchmark agent workloads multi-turn
- Don't trust single-turn benchmarks; measure with your own traces including long context and prefix cache hits.
- Replay real agent logs against candidate providers before choosing.

Try It

copy & run

# Paste into Claude or ChatGPT (a simple measurement version of the article's viewpoint)

You are an LLM inference performance engineer. Write a Python script that measures per-user speed (tok/s/user) of my API endpoint at several concurrency levels.
- Endpoint: {{API endpoint or SDK}}
- Model: {{model name}}
- Input about {{8000}} tokens, output about {{1000}} tokens
- Concurrency 1, 8, 32, with {{20}} requests each
- Stream responses and record time to first token (TTFT) separately from tok/s afterward
- Print a table of median and p95 per concurrency

After the script, give a 3-line checklist for interpreting the table.

Next to try: change the script to multi-turn with long history and compare prefix cache effects.

Terminology

ASICA chip built for one specific job. If a GPU is a universal toolkit, an ASIC is a custom-made tool.

STP/MTPSTP predicts one token at a time and MTP predicts several at once. It is like typing letter by letter versus accepting multi-word autocomplete.

Speculative decodingA small model drafts and a large model verifies to speed up output. Like an intern drafting and a senior reviewing.

Prefill-decode disaggregationRunning prompt reading and answer generation on separate chip pools. Like splitting prep and cooking across kitchen stations.

HBM4A high-bandwidth memory standard stacked next to the chip. It widens the road between warehouse and workbench.

TCOTotal cost of ownership: hardware, power, and operations. Like counting fuel and maintenance, not just a car's price.