Can gzip be a language model?

Who Should Read

A backend developer who uses LLMs but finds the compression-equals-prediction idea unfamiliar while building text generation or classification logic

Core Mechanics

• A compressor hides a probability model
  • It spends few bytes on data it expects and many on data it doesn't.
  • Bits needed for a symbol is -log2 p, where p is the probability the model assigns.
• gzip length is the score
  • DEFLATE turns repeats within a 32 KiB sliding window into cheap back-references.
  • Score is len(gzip(context + candidate)); smaller means more predicted.
  • Putting a corpus in the window makes corpus-like continuations compress small.
• Greedy byte picking fails
  • gzip only gives integer byte lengths, so adding one byte often leaves the length unchanged.
  • Many candidates tie and the signal drowns in quantization noise.
• Beam search fixes it by looking ahead
  • Extend each candidate by every byte in the corpus and keep the beam_width most compressible.
  • Repeat for horizon bytes, then commit the best span.
  • With positive temperature, sample among the finalists.

Community Reactions

• Swapping in bzip2 and zstd gave gibberish, per a commenter
  • They had MiMo-V2.6-Flash fork the code; bzip2 produced alternating-symbol runs like yxyxyx.
  • zstd produced mostly whitespace, since repeated-byte runs are nearly free.
• Skeptics say the search quality is unknown
  • The search space is far larger than what beam search covers, so results are only a lower bound on gzip as a plausibility test.
  • They note the post doesn't discuss how close beam search gets to the global optimum.
• Commenters pointed to existing gzip uses
  • One described classifying text by appending a test file to topic documents and picking the smallest gzip -9 output.
  • Others linked an earlier gzip-for-ML paper (arxiv 2212.09410) and a tsoding implementation video.
• Some pushed back on overclaiming
  • One opinion says n-gram-style models have historically been oversold as close to large neural nets.
  • Others disliked that the article seems AI-written and uses zlib rather than actual gzip.

How to Apply

• Try gzip -9 topic classification
  • Today, prepare 2-3 topic sample documents and the file to classify.
  • Append the test file to each, run gzip -9, and pick the topic with the smallest output.
  • Keep the topic documents similar in size.
• Build a zlib candidate-scoring function
  • Compute len(zlib.compress(context + candidate)) for each continuation.
  • If many scores tie, compare multi-byte spans instead of single bytes.
• Clone gzipt and swap in your corpus
  • Clone github.com/nathanrs/gzipt and pass your own domain text via --corpus.
  • Vary beam_width and horizon and compare outputs.

Try It

# Run in a Python shell (simplified version of the article's method: score candidates by zlib length)

import zlib

corpus = open("{{corpus.txt}}", "rb").read()[-30000:]
prompt = b"{{prompt}}"
cands = [b"{{candidate1}}", b"{{candidate2}}", b"{{candidate3}}"]

# Shorter means the corpus 'expected' that continuation more
for c in cands:
    print(len(zlib.compress(corpus + prompt + c, 9)), c)

# Next: extend to multi-byte spans and keep only the top few, i.e. a small beam search

Terminology

Related Papers

• The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge

  Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.

• 90% of US working hours are spent on tasks not covered well by AI benchmarks

  Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage

• Postgres 19 delayed after AI reviewers find more bugs than humans can fix

  LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted

• Show HN: Gemini 4 Argon plays StarCraft

  LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.

• A Plausible Explanation Is Enough to Mislead Multimodal Judges

  Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.

• Neural networks are clean

  Looking at neurons directly shows most have one main job, challenging superposition.

Related Resources

• https://nathan.rs/posts/gzip-lm/
• https://github.com/nathanrs/gzipt
• https://arxiv.org/abs/2309.10668
• https://arxiv.org/abs/2212.09410
• https://bellard.org/ts_zip/
• https://arxiv.org/abs/2506.01084
• https://www.youtube.com/watch?v=l6DKRf-fAAM
• https://www.youtube.com/watch?v=9n39SbRPXKQ
• https://jakobs.dev/solving-mnist-with-gzip/