Homebench – Benchmark local LLMs for speed, memory, and quality

Who Should Read

A developer with several Ollama or LM Studio models pulled who can't tell which one is actually good on their laptop

Core Mechanics

• One command measures speed, memory, and quality
  • Run pip install homebench then homebench to discover installed models and show a leaderboard.
  • It supports Ollama, LM Studio, llama.cpp, vLLM, and OpenAI-compatible servers.
  • No config, API keys, or cloud needed.
• Existing tools covered only half
  • llama-bench measures speed only.
  • lm-evaluation-harness measures quality but isn't built around local runners.
  • homebench does both locally in one pass.
• Quality grading is deterministic
  • 31 tasks are graded by exact number, multiple choice, substring, valid JSON, or regex.
  • Temperature 0 and a fixed seed make runs reproducible.
  • The optional LLM judge (--judge) is noisy by the author's own account.
• Defaults favor a quick first look
  • Default is the 3 smallest models and a fast subset of about 8 tasks.

Community Reactions

• Reports that the repo and account were deleted
  • Several commenters said the link returns 404 and the author's whole account appears gone.
  • One warned that people who already ran it should check the code and dependencies.
• The author named the methodology's weak spots
  • The author said choosing the tok/s denominator and measuring memory were hard.
  • Ollama server-side timing and client-side timing are not strictly comparable, they said.
  • The 31 tasks are a smoke test for whether quantization broke a model, they said.
• Another user had integrated similar benchmarks
  • One commenter has built local and online benchmarks into OrcaBot, including a SlopCodeBench template.
  • They were unsure how much interest there is in this space.

How to Apply

• Run a quick comparison on your models
  • Within 30 minutes, run homebench --no-tui with Ollama running.
  • Compare quality, tok/s, TTFT, and Memory for the 3 models within the same run only.
• Build a task pack from your own prompts
  • Turn 5-10 real prompts into a YAML pack with contains_any or valid_json graders.
  • Run homebench --tasks mypack.yaml to get a score on your own workload.
• Diff before and after a quantization change
  • Run with --label "before" prior to the change and again after.
  • Use homebench diff to see the quality and speed deltas.
• Check hardware fit before pulling a model
  • Run homebench fit before downloading a new model to see which quant fits.
  • Add --context 8192 if you need a larger context.

Try It

# Run in a shell (Ollama running, Python 3.9+ required)

pip install homebench

homebench doctor

homebench --no-tui --label "baseline" --md results.md

# Next: change your setting, rerun, and see the difference
homebench --no-tui --label "after"
homebench diff

# Run with your own tasks: after writing mypack.yaml
homebench --tasks mypack.yaml

Terminology

Related Papers

• The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge

  Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.

• 90% of US working hours are spent on tasks not covered well by AI benchmarks

  Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage

• Postgres 19 delayed after AI reviewers find more bugs than humans can fix

  LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted

• Show HN: Gemini 4 Argon plays StarCraft

  LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.

• A Plausible Explanation Is Enough to Mislead Multimodal Judges

  Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.

• Neural networks are clean

  Looking at neurons directly shows most have one main job, challenging superposition.

Related Resources

• https://github.com/david-g-3654/homebench
• https://github.com/Hyper-Int/OrcaBot