Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

Who Should Read

Developers who need to deploy LLMs on edge devices such as smartphones, robots, and IoT hardware, or AI infrastructure engineers looking to reduce server costs and energy consumption.

Core Mechanics

• PrismML has released three models in the Bonsai series using 1-bit weights (with a shared 16-bit scale factor per 128-bit group, effectively ~1.125-bit), available in 8B, 4B, and 1.7B parameter sizes.
• Bonsai 8B requires only 1.15GB of memory. Compared to a typical 16-bit FP 8B model that uses around 16GB, it claims to be 14x smaller, 8x faster, and 5x more energy-efficient. It demonstrated performance comparable to existing 8B models on benchmarks including IFEval, GSM8K, HumanEval+, BFCL, MuSR, and MMLU-Redux.
• Bonsai 4B is 0.57GB and Bonsai 1.7B is 0.24GB, making them even more compact. The 1.7B model reportedly achieves 130 tokens/s on an iPhone 17 Pro Max, while the 4B model records 132 tokens/s on an M4 Pro.
• The models were developed based on research from Caltech and introduce a new metric called 'intelligence density' — defined as the negative log of the model's error rate divided by model size — an attempt to measure 'intelligence per bit' rather than raw parameter count.
• Model files are distributed in llama.cpp-compatible GGUF format. Community-reported benchmarks on an RTX 3090 show the 8B model using only 4GiB of VRAM, achieving approximately 190 tokens/s with 700-token input and approximately 135 tokens/s with 6400-token input.
• The community has raised suspicions that the model may be based on Qwen3 with custom quantization kernels applied. Comments pointed out that the white paper only compares against full-precision models, without comparison to other quantized models (e.g., INT4).
• The models are described as designed for robotics and real-time agent use cases. On-device execution is available via an iPhone app (Locally AI), and tool use integration with code editors like Cursor has been confirmed to work.

Evidence

• "One user tested tool use by connecting Bonsai 8B to Cursor — the Monte Carlo pi simulation logic was correct, but UI generation failed and unnecessary symbols remained, requiring manual fixes. The general consensus was 'impressive for its size, but not perfect.' | A user who ran a SQL debugging agent benchmark reported 8 correct answers out of 25 (17 errors). It slightly outperformed Qwen3.5-4B (7/25) and fell slightly short of Nanbeige4.1-3B (9/25), but completed the full test in 200 seconds — far faster than Qwen3.5 (976s) or Nanbeige (2000s+). Granite 7B 4bit matched the speed at 199 seconds but had much lower accuracy (4/25). | A key debate was whether the model was 'trained from scratch' targeting 1-bit (like Microsoft BitNet), or was post-training quantized from an existing float model — the latter would make it just aggressive quantization. The white paper's lack of comparison against other quantized models with the same memory footprint (e.g., quantized Qwen3) was flagged as suspicious. | On a CPU-only 2018 laptop using a basic llama.cpp fork, throughput was only 0.6 tokens/s, but a user confirmed this was due to missing AVX2 optimization; after manually adding it, speed jumped to 12 tokens/s. The absence of AVX2 support in the official CPU kernel signals incomplete optimization. | A Harry Potter knowledge test produced confidently wrong answers — claiming 'Sirius Black is James Potter's father' and 'James Potter is Harry's uncle and Luna Lovegood's brother.' This serves as a warning about hallucinations in tasks requiring factual accuracy."

How to Apply

• "If you want to add on-device AI features to a smartphone app, you can download Bonsai 1.7B (0.24GB) via the Locally AI app (iOS) from its settings. There are real-world reports of it running at about half the reading speed even on an older iPhone SE2, so try it out and verify whether the accuracy is within acceptable range. | If you're running a llama.cpp-based local server, you can download Bonsai-8B.gguf in GGUF format and load it directly onto your existing server. On an RTX 3090, it uses only 4GiB of VRAM and can handle 5 concurrent requests with the --parallel 5 and --cont-batching options, letting you immediately measure GPU cost savings. | If you want to use a lightweight model for preprocessing steps (typo correction, input sanitization) in a RAG pipeline or agent, consider using Bonsai 1.7B or 4B as a lightweight filter. One user reported good results testing it on correcting NYT article paragraphs. | Before adopting this model as your primary inference model, be sure to run your own benchmarks on your specific tasks — especially multi-step reasoning and fact-verification workloads. The official benchmarks only compare against full-precision models and lack comparisons with INT4-quantized models of the same memory footprint, so you need to verify the actual advantage yourself."

Code Example

# Example of running Bonsai 8B with llama.cpp server (shared by community)
./build/bin/llama-server \
   -m ../Bonsai-8B.gguf \
   -ngl 999 \
   --flash-attn on \
   --host 0.0.0.0 \
   --port 80 \
   --ctx-size 65500 \
   --batch-size 512 \
   --ubatch-size 512 \
   --parallel 5 \
   --cont-batching \
   --threads 8 \
   --threads-batch 8 \
   --cache-type-k q4_0 \
   --cache-type-v q4_0 \
   --log-colors on

# Results: ~4GiB VRAM usage on RTX 3090
# 700-token input → ~190 tokens/s
# 6400-token input → ~135 tokens/s

Terminology

Related Resources

• PrismML Official Site (Bonsai Model Download)
• Bonsai White Paper (GitHub)
• SQL Debugging Benchmark Results (Community Shared)
• Locally AI (iOS App)
• Community Usage Test Results on Twitter