Getting 50 GB/S Back from the Apple Neural Engine

TL;DR Highlight

Splitting 1 MiB ANE weight transfers lifted Llama 3.2 1B on M3 from 10 to 24.3 tok/s

Who Should Read

A developer running on-device LLMs on Apple M3 with ANEMLL who is puzzled by lower-than-expected token throughput

Core Mechanics

1 MiB multiples crash bandwidth
- On the M3 Neural Engine, DRAM weight streaming falls to 17–19 GB/s when per-core weight bytes are an integer multiple of 1 MiB.
- Normal throughput is 45–60 GB/s, and 7 of ANEMLL's 15 models are affected.
D=2048 is far slower than D=2016
- On an M3 Air over 40 runs, D=2016 reached 44.5 GB/s while D=2048 reached 16.93 GB/s.
- Only address and size changed in the register file, and transfers still complete correctly, so it is a performance problem.
Not core count, not address layout
- Latency was the same with 1 to 16 active cores, so the throttle exists at the per-core level.
- Randomly scrambling addresses gave medians of 31.37 vs 32.29 GB/s, which ruled out DRAM bank conflicts.
1 MiB per core triggers it for any D and N
- Sweeping D and N inversely to keep 1 MiB per core still gave 17 GB/s.
- 1 MiB is 0x4000 64-byte lines, and recovery happens within ±256 lines, which is one 16 KiB page.

Community Reactions

Not all chips are affected
- A comment shared a link saying M1 and M5 Max are fine.
Question about software fixes
- One commenter asked whether M3 users are stuck unless apps work around it.
- They found the 10 to 24 tok/s jump from a workaround remarkable.
Is the RTL real or hypothetical
- A commenter asked whether the SystemVerilog is the author's guess or actual RTL source.
Terminology objection
- Someone argued 'erratum' usually means a correction notice for a document, so it may not fit here.

How to Apply

Compute per-core weight size and find 1 MiB multiples
- Today, calculate Cin×Cout×2 bytes÷16 for each projection and check for exact 1 MiB multiples.
- Llama 3.2 1B's 2048×8192 layers are examples.
Split 1 MiB-multiple projections into several tasks
- At compile time, break any kernel DMA task that is 1 MiB per core into chunks of 512 KiB or less.
- The article splits Llama 3.2 1B into 4 chunks of 0.5 MiB per lane.
Compare tok/s and GB/s before and after under equal conditions
- Alternate original and split runs on the same device and thermal state.
- Include a non-1 MiB layer as a control to confirm the effect.
Use padding only when splitting is hard
- Moving the transfer size at least ±8 KiB (256 lines) from 1 MiB restores speed.
- It changes the computation graph, so try splitting first.

Try It

copy & run

# Paste into Claude or ChatGPT (simplified version of the article's method: a model-size check)

I run LLMs on the Apple M3 Neural Engine. The ANE reads weights across 16 cores, and a reported issue slows DRAM reads to 17-19 GB/s when per-core weight bytes are an exact integer multiple of 1 MiB. Normal speed is 45-60 GB/s.

For the projection list below, compute bytes per core as (Cin × Cout × 2 ÷ 16). Tabulate the entries that are integer multiples of 1 MiB (1048576 bytes), and suggest how many chunks of 512 KiB or less to split each into.

{{projection list: name, Cin, Cout}}

Next: write a measurement plan to compare tok/s before and after splitting under identical conditions.

Terminology

ANE (Apple Neural Engine)A dedicated neural network unit inside Apple chips. Think of it as a specialized calculator for matrix math during inference, instead of the GPU.

DRAM weight streamingContinuously reading model weights from memory to feed computation. Each token touches all the weights, so decode speed depends heavily on it.

PrefetchRequesting data before it is needed. It's like a kitchen pulling ingredients before the customer orders.

ErratumA known flaw in a chip design. Here it refers to a performance flaw in the M3 ANE's RTL (hardware design).

Kernel DMAThe ANE engine that moves weights from memory into the cores without CPU involvement. Here 'kernel' means convolution kernel, i.e. a block of weights.