Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

TL;DR Highlight

Streaming expert weights from SSD lets an 80B Qwen run in 4.3 GB of RAM on a Mac.

Who Should Read

An iOS/macOS developer who wants local LLMs in a Mac or iPhone app but is blocked by RAM limits.

Core Mechanics

80B runs in 4.3 GB of RAM
- Qwen3-Next-80B-A3B 4-bit is 42 GB on disk, 4.3 GB peak RAM, 4.5 to 5 tok/s decode on an M5 Mac.
- Qwen3.6-35B-A3B 4-bit is 18 GB on disk, 2.6 GB RAM, 7 to 11 tok/s.
The trick is expert streaming
- MoE (a model that uses only some expert blocks per token) activates only about 3B parameters per token.
- Only the small dense core stays in memory, and experts are read from SSD on demand.
One SSD read per expert
- Experts are repacked into fixed-stride blobs in a .qpack container so one expert is one pread.
- No mmap is used, and hot experts sit in an LFU plus recency cache.
Models bigger than RAM still run
- On an M4 Max 64 GB, the 8-bit 80B (78.8 GiB of weights) runs in 6.9 GiB at about 4.8 tok/s.
- The 4-bit 397B (207.6 GiB) runs in 12.6 GiB at about 1.4 tok/s.
Speed follows active parameters

Community Reactions

Prefill is the bottleneck, critics say
- One commenter noted that half an hour for 10k tokens on an M5 means decode numbers don't tell the whole story.
- Others criticized disk-swapping approaches as slow and hard on drives.
Interest in tuning RAM use
- A commenter with a 32 GB MacBook asked whether RAM use is tunable for more speed.
- Another planned to test a larger RAM cache on a 24-32 GB Mac.
Other platforms and prior work
- Asked about Android/Linux/Windows, a commenter pointed to BigMoeOnEdge.
- Another noted ANEMLL already ran a 400B-class model on an iPhone with similar weight streaming.
Response from TurboFieldfare's author
- The TurboFieldfare author thanked the project for crediting it.
- Another commenter asked how local models can handle web access and external services.

How to Apply

Run the 35B on an Apple Silicon Mac
- Clone the repo, run swift build -c release, and download the 18 GB 4-bit 35B container with swiftlet-repack.
- Chat with swiftlet chat and measure speed and RAM yourself.
Point existing code at swiftlet-server
- Change your OpenAI-compatible client's base URL to http://localhost:8080/v1.
- Swap an internal tool or prototype to the local model and compare quality.
Add SwiftletCore to your app
- For macOS/iOS apps, use SwiftletSession from the Swift package for streaming chat.
- Rely on its built-in memory-pressure handling and conversation caching.
Measure long-prompt workloads first
- For agents with long system prompts, time prefill on your real prompt.
- On a 16 GB device, start with --cache-gb 2.

Try It

copy & run

# Paste into a terminal (Apple Silicon Mac, macOS 14+, 18 GB+ free SSD)

git clone https://github.com/leonickson1/Swiftlet.git && cd Swiftlet
swift build -c release

# Download the 35B 4-bit container
.build/release/swiftlet-repack \
  --from-url https://pub-c0cfece2dbc340dbb2cd9d94310a7d68.r2.dev/qwen3.6-35b-qpack \
  --output ~/models/qwen3.6-35b.qpack

# Run a chat
.build/release/swiftlet chat ~/models/qwen3.6-35b.qpack \
  "What is the capital of Spain?" "And what about France?"

# Next: serve it as an OpenAI-compatible server
.build/release/swiftlet-server --model ~/models/qwen3.6-35b.qpack --port 8080

Terminology

MoE (Mixture-of-Experts)A model with many experts where only a few are used per query. The model is big but each step computes little.

expert streamingKeeping rarely used expert weights on SSD and loading them only when needed. Like pulling just the page you need instead of spreading the whole book on your desk.

prefill / decodePrefill processes your input prompt; decode generates the answer one token at a time.

Gated DeltaNetA linear attention that summarizes context in a fixed-size state. Those layers need no growing KV cache at long context.

preadA system call that reads from a given file offset directly. Unlike mmap, it avoids page-cache thrash.