Sub-1-Bit LLM Compression via Latent Factorization

Who Should Read

An inference-optimization engineer exploring extreme quantization to run large LLMs on local GPUs

Core Mechanics

• Factor, then binarize the weights
  • Each dense weight matrix is split into low-rank latent factors.
  • The factors are binarized and magnitude is restored with lightweight learned scales.
• Goes down to 0.1 bits per weight
  • Designed for the 1.0 to 0.1 bits-per-weight range.
  • The original model architecture is preserved at inference.
• LittleBit-2 changes only initialization
  • It rotates SVD-derived latent factors to align with the binary hypercube (Joint-ITQ).
  • There is no extra inference overhead and the deployed factorized layer stays the same.
• It requires training
  • It uses QAT (retraining with quantization in the loop), SmoothSign, and optional residual factorization.
  • The example trains Llama-2-7b on c4_wiki for 5 epochs.
• Supported models and license

Community Reactions

• Concern about large quality loss
  • One commenter said performance drops from 80% to 47% on Wikitext-2.
  • They noted FP4 approaches at a 4.25-4.5 bit budget keep 80% and were not compared.
• Hybrid approach suggested
  • A commenter suggested using sub-bit only where it is least needed, such as later layers.
  • Another idea was to approximate a 4-bit matmul with several 1-bit operations.
• Maybe the wrong axis
  • Commenters said MoE expert offloading and NVMe streaming have been more impactful for local AI.
  • Another said 1-bit quants have been less useful than smaller models using the same memory.
• No real-world reports yet
  • Commenters asked whether pretrained checkpoints or Apple Silicon benchmarks exist.
  • One said large quality loss in the paper does not make it a negative result.

How to Apply

• Measure the quality drop with eval.py today
  • If a LittleBit checkpoint exists on the Hugging Face Hub, pass it via --model_id and measure wikitext2 perplexity.
  • Compare it against your current 4-bit model under identical conditions.
• Run one QAT job on Llama-2-7b
  • Run the README single-GPU command as is and lower --eff_bit from 1.0 step by step while logging quality.
  • Add --use_itq True to compare against SVD-only initialization.
• Check the license before production use
  • CC BY-NC 4.0 means do not use it directly in commercial services.
  • Limit usage to internal experiments and memory-budget studies.

Try It

# Run in a terminal (GPU, Python 3.12, after the README setup steps)

pip install "transformers==4.51.*"

CUDA_VISIBLE_DEVICES=0 python -m main \
  --model_id meta-llama/Llama-2-7b-hf \
  --dataset c4_wiki \
  --save_dir ./outputs/Llama-2-7b-LittleBit-2 \
  --num_train_epochs 5.0 \
  --per_device_train_batch_size 4 \
  --lr 4e-05 \
  --warmup_ratio 0.02 \
  --report wandb \
  --quant_func SmoothSign \
  --quant_mod LittleBitLinear \
  --residual True \
  --eff_bit 1.0 \
  --kv_factor 1.0 \
  --min_split_dim 8 \
  --l2l_loss_scale 10.0 \
  --use_itq True

# Evaluate after training
CUDA_VISIBLE_DEVICES=0 python eval.py \
  --model_id ./outputs/Llama-2-7b-LittleBit-2 \
  --seqlen 2048 \
  --ppl_task wikitext2,c4

# Next: drop --use_itq True, rerun with the same settings, and compare perplexity

Terminology

Related Papers

• Mercury 3: Diffusion at scale

  Scaling diffusion LLM 40x gives Mercury 3: 4–8x faster than speed-tier models, similar quality

• Timestral: Mistral's time-series foundation models

  A 209M time-series model ranks first on three benchmarks and runs on CPU.

• DeepSeek V4.1 Flash 1017 (Max)

  DeepSeek V4.1 Flash 1017 Max scores 45 on the Intelligence Index at 211 tok/s and $0.30 input.

• If you don't run your own LLM proxy, how do you know whats going on?

  Running your own LLM proxy shows which keys, models, and cache misses drive your usage.

• DeepSeek: DeepSeek v4.1 Flash 1017

  DeepSeek V4.1 Flash 1017: $0.05/M input, 1M context, built for agents

• DeepSeek-v4.1-Flash-1017

  DeepSeek V4.1 Flash minor update lifts coding and agent benchmarks while cutting run time and output length

• Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware

Related Resources

• https://github.com/SamsungLabs/LittleBit