TurboQuant: Redefining AI efficiency with extreme compression
TL;DR Highlight
Google Research 2-stage vector compression — PolarQuant + QJL achieves 6x KV cache reduction with zero accuracy loss and 8x attention speedup on H100 GPUs
Who Should Read
ML engineers optimizing LLM inference cost and latency; teams battling KV cache memory bottlenecks in long-context services
Core Mechanics
- PolarQuant: randomly rotates vectors then converts to polar coordinates, mapping angle patterns onto a fixed circular grid — eliminates quantization constant storage overhead entirely
- QJL (Quantized Johnson-Lindenstrauss): uses just 1 additional bit to detect and correct remaining compression errors — reduces each vector number to a single sign bit (+1/-1)
- 6x KV cache size reduction, 3-bit quantization, zero training required — zero accuracy loss across all benchmarks
- Up to 8x performance increase for attention computation on H100 GPUs vs 32-bit unquantized baseline
- Superior recall vs PQ and RabbiQ in vector search — applicable to large-scale ANN search as well
Evidence
- H100 GPU benchmark: up to 8x attention computation speedup vs 32-bit unquantized, zero accuracy loss across all downstream benchmarks
- Independent llama.cpp and PyTorch implementations published immediately (github.com/mudler/llama.cpp, github.com/tonbistudio/turboquant-pytorch)
How to Apply
- Apply TurboQuant to long-context LLM services where KV cache memory is the bottleneck — expect 6x memory reduction and 8x attention speedup
- Experiment immediately with the llama.cpp integration — no training required, drop-in for existing models
- Also applicable to vector DB (ANN search) performance improvement — better recall than PQ
Terminology
Related Papers
Training an LLM in Swift, Part 1: Taking matrix mult from Gflop/s to Tflop/s
Apple Silicon에서 Swift로 직접 행렬 곱셈 커널을 구현하며 CPU, SIMD, AMX, GPU(Metal)를 단계별로 최적화해 Gflop/s에서 Tflop/s 수준까지 성능을 높이는 과정을 상세히 설명한 글이다. 프레임워크 없이 LLM 학습의 핵심 연산을 밑바닥부터 구현하고 싶은 개발자에게 Apple Silicon의 성능 한계를 체감할 수 있는 드문 자료다.
Removing fsync from our local storage engine
FractalBits가 fsync 없이 SSD 전용 KV 스토리지 엔진을 구현해 동일 조건 대비 약 65% 높은 쓰기 성능을 달성한 설계 방법을 공유했다. fsync의 메타데이터 오버헤드를 피하기 위해 사전 할당, O_DIRECT, SSD 원자 쓰기 단위 정렬 저널을 조합한 구조가 핵심이다.
Google Chrome silently installs a 4 GB AI model on your device without consent
Google Chrome이 사용자 동의 없이 Gemini Nano 4GB 모델 파일을 자동 다운로드하고, 삭제해도 재다운로드되는 문제가 발견됐다. GDPR 위반 가능성과 수십억 대 기기에 적용될 때의 환경 비용 문제가 제기되고 있다.
How OpenAI delivers low-latency voice AI at scale
OpenAI redesigned its WebRTC stack to serve real-time voice AI to over 900 million users, detailing the design decisions and trade-offs of a relay + transceiver split architecture.
Efficient Test-Time Inference via Deterministic Exploration of Truncated Decoding Trees
Deterministic Leaf Enumeration (DLE) cuts self-consistency’s redundant sampling by deterministically exploring a tree of possible sequences, simultaneously improving math/code reasoning performance and speed.