I Tried Using Compressors as Language Models
Nathan Barry wrote a fun post asking whether gzip can be a language model. The trick is simple: prime the compressor with a corpus, append a candidate continuation, and prefer the continuation that makes the compressed output shortest. If a continuation looks like the corpus, the compressor can encode it cheaply.
I wanted to check whether this was actually true, then try the same decoding idea with other compressors. The short answer: yes, gzip can generate text, if "generate" means "beam search over byte strings scored by compressed size." It is not good in the modern LM sense. It copies and recombines local patterns. But it is clearly not random.
Setup
I used Tiny Shakespeare as the corpus and this prompt:
MENENIUS:
For gzip validation I used the same broad settings as the original repository: 200 generated bytes, horizon 24, beam width 32, temperature 0.5, tail 80, and a 30,000 byte corpus window. For the cross-compressor comparison I used a smaller shared search budget because bzip2, brotli, xz, zstd, and lz4 do not expose the same cheap zlib state-copy trick in this implementation.
Gzip Worked
Here is my gzip run. It is not coherent Shakespeare, but it has speaker labels, line breaks, words, and local phrase fragments:
MENENIUS: 'Tho AUFIDIUS: We hate alike: Not MENENI MENENI MENENI MENENI MEe And that his country'sHadst thou beheld-- MARA carbuncle entire, as big as tho ENI MEe And q
That is enough to validate the claim. The compressor is acting like a crude byte-level predictive model, and beam search turns that scoring function into generated text.
Other Compressors
| Compressor | Result | My read |
|---|---|---|
| gzip | Shakespeare-like fragments, but repetitive | Works. Mostly local copying and recombination. |
| zlib | Matched gzip in the budgeted run | Expected: same DEFLATE family, different wrapper. |
| brotli | Best sample in the small comparison | Surprisingly text-like under this search budget. |
| bzip2 | Mostly character soup | It found compressible byte patterns, not readable text. |
| zstd | Whitespace and punctuation heavy | Fast, but poor as a generator here. |
| lz4 | Repeated punctuation and spacing | Very local matching shows up as very local repetition. |
| xz | Punctuation and whitespace | I had to reduce the search budget to keep runtime sane. |
The brotli sample was the most readable:
MENENIUS: O, true-bred! First Senator: Your company to the Capitol; where, I know, Our greatest friends atte t t b o a ,
The zstd and lz4 samples were much less impressive. They were technically optimized by the same rule, but the outputs looked like this:
MENENIUS: , l . - , n
Conclusion
The interesting part is not that gzip is secretly a good language model. It is not. The interesting part is that compression gives you a real predictive signal without training a model. If a byte sequence compresses well in context, the compressor is telling you that sequence was unsurprising according to its internal matching machinery.
My ranking from this quick experiment: brotli first, gzip/zlib second, then zstd, xz, lz4, and bzip2 far behind. I would not take that as a universal compressor ranking. It is a ranking of this exact corpus, prompt, decoder, and search budget.
The source experiment is intentionally small: a Python script, Tiny Shakespeare, and a beam search over byte strings. The result is a nice reminder that "prediction" and "compression" are closer than they feel when we only think about modern neural networks.