Breaking the 1.58-bit Barrier for Ternary LLMs

TL;DR Highlight

Exploiting 51.5% zeros, BITCOS stores ternary weights at 1.485 bits and speeds decoding up to 1.27x.

Who Should Read

Inference engineer running ternary-quantized LLMs on edge or on-device hardware who is struggling with memory footprint and decode speed

Core Mechanics

Ternary models were using 1.625 bits each
- Ternary LLMs store each weight as one of {-1,0,+1}.
- The information-theoretic floor is log2(3)≈1.585 bits.
- The common format, five-trit packing (5 weights per byte), actually costs 1.625 bits.
The three values are not equally likely
- Across 29 ternary LLMs, zeros made up up to 51.5% of all weights.
- Existing packing assumes -1, 0, and +1 are equiprobable.
BITCOS is a bitmap plus sign vector
- A dense presence bitmap marks which weights are nonzero.
- Signs of only the nonzero weights are stored in a compacted sign vector.
- With zero density z, it costs 2−z bits per weight.
Smaller in 26 of 29 models
- BITCOS stores weights more compactly than five-trit packing in 26 models.
- On the sparsest model it reaches 1.485 bits per weight.

Community Reactions

Surprise that zeros weren't already exploited
- One commenter assumed models already worked this way and even explained it so to a friend.
- They asked why an adaptive design wasn't used from the start.
Doubts about use in memory
- One asked whether this only helps file size, since memory would need expanding back to 5 trits per byte.
- Another was surprised a variable-length encoding works directly as an in-memory format.
Skepticism and hardware hopes
- One commenter argued ternary quantization makes little sense, saying vector quantization and trellis methods are better for PTQ.
- Others thought it would fit ASICs well for on-device power efficiency.
Alternative encodings and edge use
- Someone suggested arithmetic coding to squeeze out a few more centi-bits.
- Another suggested an 8-bit LUT over common 6-trit vectors.
- People running LLMs on limited-RAM edge devices, or fitting models into 16GB VRAM, found it relevant.

How to Apply

Measure the zero ratio of your ternary model first
- Read the weight tensors of your ternary checkpoint and count -1, 0, and +1.
- Compute 2−z bits from the zero density z and compare it with 1.625 bits.
Pick high-z models as adoption candidates
- BITCOS beat five-trit in 26 of 29 models, so check whether z exceeds 0.375.
- Once you have a candidate, check whether your hardware supports AVX-512, AVX2 or Xe2.
Set expected speedups per target hardware
- Use up to 1.18x for CPU decode and up to 1.27x for GPU as the baseline.
- These figures come from the paper's 5 platforms, so measure decode throughput in your own environment.

Try It

copy & run

# Python (numpy, pip install numpy) - compute zero density and BITCOS bits for your ternary weights

import numpy as np

# {{w}}: replace with your array of -1/0/+1 weights (random sample data here)
w = np.random.choice([-1, 0, 1], size=1_000_000, p=[0.245, 0.51, 0.245])

z = float((w == 0).mean())
print(f"zero density z = {z:.3f}")
print(f"BITCOS: {2 - z:.3f} bit/weight")
print("five-trit: 1.625 bit/weight")
print("BITCOS is smaller" if 2 - z < 1.625 else "five-trit is smaller")

# Next: compute z per layer on a real checkpoint

Terminology

ternary LLMAn LLM whose weights take only -1, 0, or +1. Think of rounding every number to one of three notches.

five-trit packingPacking five 3-valued weights (trits) into one byte, since 3^5=243 fits in 256. It effectively costs 1.625 bits per weight.

presence bitmapA table with one bit per position marking whether it is nonzero. Like an attendance sheet marking who showed up.

sign vectorAn array holding only the +/- signs of the nonzero weights. Like a list of just the people who attended.

unpackingTurning packed weights back into a form usable for computation. Like opening a parcel to take the items out.

decode throughputHow many tokens per second are generated in the step-by-step generation phase.