Show HN: Cactus Needle 3: 8-29MB automation models can match DeepSeek V4 Flash

TL;DR Highlight

An 8–29MB model beats 10x larger ones on mobile tool calls, but real-world use is still rough.

Who Should Read

App or embedded developers who want offline tool calling or structured extraction in smart home or mobile products.

Core Mechanics

One 8–29MB binary
- Needle 3 targets phones, wearables, robots, smart home, cars, and microcontrollers.
- It has 29M–121M parameters across 2–20 layers.
- It gives up general chat ability to focus on tool calls and extraction.
Beats 10x larger models on tool calls
- It claims better accuracy than models 10x its size on mobile tool calls.
- On extraction it matches models 2–3x larger.
- Baselines ran at f16 on vLLM, and DeepSeek V4 Flash ran through its cloud API.
Pick the layer count you need
- Every layer is a sub-network of increasing capacity, so you can choose 2L to 20L.
- This is called intelligence laddering, with CQ2-bit binaries of 9–29MB.
- On a Raspberry Pi 5 it decodes at 400–4k tok/s and prefills at 1–10k tok/s.
Fine-tuning passes DeepSeek V4 Flash
- Fine-tuning on DroidCall lifts every sub-network by 18–36 points.

Community Reactions

Indirect smart-home phrases often failed, per user reports
- One user said "it's too cold" lowered the thermostat, and "make it hot" also lowered it from 20 to 18 with high confidence.
- Another said "warm the house" set lights to a warm brightness instead.
- Direct commands like "turn all the lights on/off" worked.
Reasoning field seems untrustworthy, some said
- In the car demo, "turn it up" was interpreted as lowering the volume.
- In the house demo, the reasoning correctly wanted the alarm off but the action did not happen.
- "My car crashed I need help" picked play_music with confidence 1.
FunctionGemma beat it on a Runescape DB tool task, one user found
- Exact-argument match was 85.2% for FunctionGemma BF16, 20.4% for Needle 3 W4A8, and 18.7% for Needle 2 W4.
- They said it improves on Needle 2.
- Another user found MNLI far better for labeling.
Dataset quality and anti-use-cases matter, commenters argued

How to Apply

Test offline tool calling on your own 5 tools
- Install cactus-needle today and wrap 3–5 of your tools with @needle.tool.
- Run 20 real user phrases mixing direct and indirect wording, and log accuracy separately.
Branch on confidence: act, confirm, refuse
- Execute at 0.7 or higher, ask for confirmation below that, and refuse on an empty result.
- Raise the threshold or always confirm for risky devices like ovens or heating.
Narrow tool descriptions and add triggers
- Split tools to one per action and name enum values after what users say.
- Add regex triggers to intents that must always reach a tool.
LoRA-tune on your domain and compare
- Fine-tune a 4L sub-network with LoRA on your toolset and compare it to your current model on the same test set.
- Measure argument accuracy, latency, and RAM before deciding to switch.

Try It

copy & run

# Run in terminal: quick Needle 3 tool-calling test (based on the article's example)

pip install cactus-needle

# Save the below as test.py (replace with your own functions)
import needle
from typing import Literal

@needle.tool(triggers=[r"\b(turn|switch|power|flip)\b.*\b(on|off)\b", r"\btoggle\b"])
def control_device(device: str, action: Literal["on", "off", "toggle"]):
    "Switch or toggle any named smart-home device."
    return {"device": device, "action": action}

agent = needle.Needle(tools=[control_device])
r = agent.complete("{{user sentence}}")
calls = r["function_calls"]
if calls and r["confidence"] >= 0.7:
    print("execute", calls)
elif calls:
    print("confirm", calls, r["reasoning"])
else:
    print("refuse")

# Next: try indirect phrases like "it's too cold" and check whether confidence is low

Terminology

Simple Attention NetworkCactus's architecture, said to use over 2x fewer MFLOPs per token than a same-config transformer. Think of it as a lightweight transformer variant.

Intelligence ladderingEach layer is a sub-network with growing capacity, so you can cut the model at the depth you need. It's like climbing a ladder only as high as required.

CQ2-bit quantizationCompresses weights to around 2 bits to shrink the binary to 8–29MB. Like downscaling a photo to low resolution.

LoRAFine-tuning that freezes the base weights and trains small adapters. It's like leaving the book intact and adding sticky notes.

Confidence floorA cutoff (0.1) below which calls are withheld. It's the minimum ID check at the door.