Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots

TL;DR Highlight

An 8–29MB on-device model claims to beat 10x larger models on mobile tool calls.

Who Should Read

An app developer who wants to handle tool calls on-device for smart home or mobile apps without cloud LLM calls.

Core Mechanics

Needle 3 is a single 8–29MB binary
- It targets mobile, wearables, robots, smart home, automotive and microcontrollers.
- It has 29–121M parameters and 2–20 layers.
It does only three things
- Tool calls, structured extraction and text embedding.
- It gives up general chat ability to focus on these.
Claims to beat 10x larger models on mobile tool calls
- The post says it matches 2–3x larger models on extraction.
- Baselines ran at f16 under vLLM, and DeepSeek V4 Flash through its cloud API.
Pick a size from the layer ladder
- Every layer is a sub-network, selectable from 2L to 20L.
- CQ2 (2-bit) quantisation gives 9–29MB binaries.
Confidence score splits execute, confirm and refuse
- Each response carries a confidence, and calls below 0.1 go to suppressed_calls.

Community Reactions

Commenters report it often misses the meaning in the demo
- One said "Make the living room dark" turned lights on and ignored brightness.
- Another said "Make it a little warmer" set mode to cool.
Commenters flag unreliable units and unknown requests
- One tried adding units to the description, and Celsius vs Fahrenheit still wobbled.
- They saw higher confidence for the wrong unit.
- Others said it should call no tool for requests it doesn't understand, but does so inconsistently.
Some doubt its use for robotics
- One user said it failed a simple coordinate navigation prompt.
- They suggested it might only help at the planning level.
Positive takes on using it as a router
- One argued sending every input to a big LLM is slow and costly, so a small model for fixed actions with fallback is good.
- Another said they already used Needle as a router.

How to Apply

Rewrite tool descriptions narrow and plain
- Today, split your tool schema into one tool per action.
- Name enum values as users say them and put value formats in descriptions as examples.
Add three-way routing on confidence
- Execute at 0.7 or above, ask the user below that, and refuse on empty results.
- Wrong calls sometimes had high confidence in the demo, so validate thresholds on your own data.
Use the small model as a router and fall back to a big LLM
- Handle fixed commands with Needle.
- Send empty results or low confidence to a cloud LLM.
LoRA fine-tune on your own tool set
- Tune on your product's tools and measure accuracy from the smallest sub-network.
- Pick a layer count that fits your device and build it with needle build --layers.

Try It

copy & run

# Run in terminal (Python 3, based on the Needle post's example)

pip install cactus-needle

# Save the following as a .py file and run it
import needle
from typing import Literal

@needle.tool(triggers=[r"\b(turn|switch|power|flip)\b.*\b(on|off)\b", r"\btoggle\b"])
def control_device(device: str, action: Literal["on", "off", "toggle"]):
    "Switch or toggle any named smart-home device."
    return {"device": device, "action": action}

agent = needle.Needle(tools=[control_device])
r = agent.complete("toggle the garage door")
calls = r["function_calls"]
if calls and r["confidence"] >= 0.7:
    print("execute", calls)
elif calls:
    print("confirm", calls, r["reasoning"])
else:
    print("cannot do")

# Next: try a vague request like "make it a little warmer" and check confidence and arguments

Terminology

tool callThe model reads a user request, picks one of your app's functions and fills in its arguments. It's like a translator from spoken commands to function calls.

intelligence ladderingA design where you can use anywhere from 2 to 20 layers of the model. Several sizes live inside one model like rungs on a ladder.

CQ2 quantisationA way to shrink the model by storing weights in 2 bits. Like saving a photo at lower resolution, you trade some precision for size.

LoRAA fine-tuning method that freezes the base model and trains only small adapter weights. Like leaving the book alone and adding sticky notes.

confidence gatingUsing the model's confidence score to choose between executing, confirming and refusing. A safety valve that asks a human when the score is low.