saved
Can gzip be a language model?
Hraness wrote this summary from a saved copy of the source. Quotations are taken word for word from the source.
gist
Nathan Barry shows that gzip can act as a crude language model when a corpus is loaded into DEFLATE’s sliding window and candidate continuations are ranked by compressed length. Because compression rewards byte sequences resembling recent context, beam search can produce Shakespeare-like fragments without neural weights or training. The output remains incoherent and quantized, but the experiment makes compression–prediction equivalence tangible and offers a tiny, inspectable generative model.
ideas
- Compression hides prediction. The bits a compressor spends encode an implicit probability model: expected continuations cost less.
- DEFLATE supplies the memory. Its 32 KiB sliding window makes corpus-like byte sequences cheap through back-references.
- Beam search makes generation viable. Looking ahead across spans avoids the ties and quantization noise of one-byte greedy scoring.
- Context must stay bounded. Keeping only a recent tail limits verbatim loops that would otherwise dominate the search.
quotes
“every prediction model is inherently a compressor, and all compression algorithms are prediction models.”
“A continuation that gzip “expected”, because it echoes text already in its window, compresses to almost nothing.”