Retrospectively Reverse-Engineering Apple's Neural Engine

TL;DR Highlight

Reverse-engineering the M1 ANE shows CNN-tuned dataflow and fixed-size command streams, explaining its limits for LLMs.

Who Should Read

App or inference developers running on-device LLMs on Apple Silicon who wonder why the ANE underdelivers.

Core Mechanics

The ANE is tuned for CNN dataflow
- The MAC unit does the same dot product for CNNs and attention.
- What specializes the ANE is the dataflow around the MACs, not the MACs themselves.
- The author says autoregressive decode broke the predictable-reuse assumption that dataflow relies on.
16 cores, 2048 MAC lanes
- Each core has 128 FP16 lanes (256 for INT8).
- 16 cores give 2048 parallel lanes.
- The accumulator is 32-bit and saturated at 2^15 in the author's test.
Activation is a 33-entry lookup table
- The compiled tanh held 33 FP16 words matching tanh(i/8) samples.
- An impulse test produced a triangle output, showing linear interpolation between entries.
- ReLU uses mode 1 with no coefficients, and the compiler folds scale and bias into the convolution weights.
The driver knows no CONV or RELU opcodes
- Operations are already compiled into a stream of task descriptors (TDs).

Community Reactions

Questions on how this relates to the M4 ANE work
- One commenter asked whether M4 and later ANEs add capabilities or are just faster versions.
- They also said the intro conflates the ANE with the M5 GPU Neural Accelerators (NAX).
- They added that M6 and A20 reportedly double the ANE blocks.
Experience porting transformers as if they were CNNs
- One commenter said the port meant using 4D tensors with sequence on the last axis and 1x1 convs instead of matmuls.
- Another reported running Qwen models on the ANE with about 4 tok/s and 15-20 tok/s decode.
Apple's upcoming Core AI framework
- A commenter noted Core AI arrives this fall and lets apps run newer architectures across CPU, GPU, and Neural Engine.
- It goes beyond what Core ML supported for PyTorch and TensorFlow models.

How to Apply

Check where your model actually runs
- Profile your Core ML model in Xcode Instruments today to see which ops run on ANE, GPU, or CPU.
- List the layers that fall off the ANE.
Try a CNN-style rewrite of a transformer layer
- Convert one attention or FFN matmul to a 1x1 convolution with sequence on the last axis.
- Compare latency before and after.
Split prefill and decode across units
- Measure the same model with ANE-only, GPU-only, and all compute units.
- Tabulate tokens/sec and pick your deployment target.
Use the prompt below to audit your model

Try It

copy & run

# Paste into Claude or ChatGPT (simplified version of the article's idea: ANE suitability check)

I'm a developer deploying an on-device model with Core ML. I read that the ANE is tuned for CNN-style dataflow and that autoregressive decode has unpredictable reuse, which hurts it.

My model structure:
{{model structure (layer types, sequence length, batch)}}

Tell me:
1. Which layers are likely hard to run on the ANE and why
2. How to convert matmuls to 1x1 convs with sequence on the last axis
3. Which metrics to measure per ANE/GPU/CPU and in what order

Next step: convert one layer as suggested and compare latency per compute unit.

Terminology

MAC (Multiply-Accumulate)A unit that multiplies two numbers and adds the product to a running sum. Like a calculator button that keeps adding products to memory.

DataflowHow and when MAC inputs and outputs enter, stay, and move. Like designing the part routes on a factory line.

Task Descriptor (TD)A fixed-size blob of register values telling the ANE what to run. More like a machine settings sheet than an instruction list.

BAR (Base Address Register)The base that relative addresses in a command stream refer to. Like fixing a building number and only writing room numbers.

LUT (Lookup Table)Precomputed function values stored in a table and looked up at runtime. Reading a value chart instead of computing a formula.