WebLLM: high-performance in-browser LLM inference engine

Who Should Read

A frontend developer who wants to run LLMs directly in users' browsers to cut server costs and protect privacy

Core Mechanics

• Runs in the browser, no server
  • Inference runs entirely inside the browser with WebGPU hardware acceleration.
  • The README says no server support is needed.
• Uses the OpenAI API style
  • You call engine.chat.completions.create and get streaming, JSON-mode, and seeding.
  • Function-calling is still marked WIP.
• Supported model families are fixed
  • Built-ins include Llama 3, Phi 3, Gemma-2B, Mistral-7B-v0.3, and Qwen2 0.5B/1.5B/7B.
  • You can also plug in custom models compiled to MLC format.
• Workers keep the UI responsive
  • Offload inference to a Web Worker or Service Worker to free the UI thread.
  • Service Workers can be killed by the browser anytime, so add error handling.
• First run needs a model download
  • The README warns the first uncached run can take significant time.

Community Reactions

• Doesn't run without WebGPU
  • One commenter got WebGPUNotAvailableError on both Firefox and Chromium on Linux.
  • Another hit a maxStorageBuffersPerShaderStage limit error (requested 10, limit 9).
• 500MB-1GB download per session
  • A commenter notes each browser session downloads roughly 500MB to 1GB depending on the model.
  • They suggest adding a disclaimer on customer-facing sites.
• Maintenance concerns and an alternative
  • One user says they ripped it out 6 months ago and calls the project de facto dead.
  • Another says it hasn't been updated since Gemma 2 and recommends Transformers.js.
• Measured speed report
  • A commenter got prefill 181.2 tok/s and decode 58.5 tok/s with a 1B model on an M4 Max in Safari.
  • They call it unimpressive for 1B but impressive for running in a browser.

How to Apply

• Check WebGPU in your browser with the demo
  • Open chat.webllm.ai in Chrome and load a small model.
  • See whether WebGPUNotAvailableError appears and whether the speed is usable.
• Pick one privacy-sensitive feature for local inference
  • Choose an input you hesitate to send to a server (e.g. summarization or classification).
  • Keep your chat.completions call shape and swap only the engine.
• Move inference into a Web Worker
  • Create the engine with CreateWebWorkerMLCEngine to separate it from the UI thread.
  • Wire initProgressCallback to a progress bar to cover the first download.
• Disclose download size and compare alternatives
  • Tell users about the 500MB-1GB download and add a server API fallback for no-WebGPU browsers.
  • If maintenance worries you, run the same task on Transformers.js and compare.

Try It

// Run in a browser project (e.g. Vite): npm install @mlc-ai/web-llm

import { CreateMLCEngine } from "@mlc-ai/web-llm";

const engine = await CreateMLCEngine(
  "Llama-3.1-8B-Instruct-q4f32_1-MLC",
  { initProgressCallback: (p) => console.log(p) }
);

const chunks = await engine.chat.completions.create({
  messages: [
    { role: "system", content: "You are a helpful AI assistant." },
    { role: "user", content: "{{question}}" },
  ],
  stream: true,
});

let reply = "";
for await (const c of chunks) {
  reply += c.choices[0]?.delta.content || "";
  console.log(reply);
}

// Next: swap the model ID for a smaller one (e.g. Llama-3.2-1B-Instruct-q4f16_1-MLC) and compare speed and download size.

Terminology

Related Papers

• Mercury 3: Diffusion at scale

  Scaling diffusion LLM 40x gives Mercury 3: 4–8x faster than speed-tier models, similar quality

• Timestral: Mistral's time-series foundation models

  A 209M time-series model ranks first on three benchmarks and runs on CPU.

• DeepSeek V4.1 Flash 1017 (Max)

  DeepSeek V4.1 Flash 1017 Max scores 45 on the Intelligence Index at 211 tok/s and $0.30 input.

• If you don't run your own LLM proxy, how do you know whats going on?

  Running your own LLM proxy shows which keys, models, and cache misses drive your usage.

• DeepSeek: DeepSeek v4.1 Flash 1017

  DeepSeek V4.1 Flash 1017: $0.05/M input, 1M context, built for agents

• DeepSeek-v4.1-Flash-1017

  DeepSeek V4.1 Flash minor update lifts coding and agent benchmarks while cutting run time and output length

• Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware

Related Resources

• https://github.com/mlc-ai/web-llm
• https://chat.webllm.ai/
• https://npm.im/webml-kit