A DISSECTION, FOUR FILE FORMATS,
AND 28% OFF THE INDEX

Dissect first, write Rust second

The size work started as an epic with a waste list written against the code: numeric doc-values stored as raw 8-byte lanes, postings block headers carrying bytes the reader could derive, per-term metadata in fixed 24-byte records, the same keyword strings stored three times per segment. A confident list. Parts of it were wrong, and the way we found out which parts is the method worth writing down.

Stage zero built a harness, benchmarks/index-size: 100,000 documents — 4,008 real telemetry events cycled 25 times, each with a timestamp, a model name, token counts, a cost, a tenant, and a body of text — bulk-indexed into a throwaway node, force-merged to a single segment, then a byte census by file family and by field. Every format change in this post landed with its A/B result committed beside it.

The rule the epic settled on: before writing a codec, copy the segment file out of the data directory and dissect it offline — python3 and the zstd command line are enough — and predict the payoff from the actual bytes. Only then open the editor. Four formats went through that gate:

Each row is a same-day A/B against a control binary on the same corpus at the balanced compression level, one force-merged segment; the first two stages' tables live in the changelog entries and the last two stages' result files are committed beside the harness. Those last two were also re-measured together on the merged tree — 4,176,306 to 3,989,254 B — and subtracting the .meta delta from that control reproduces the endpoint to the byte. Sum all four family deltas and the index that entered the epic at roughly 5.52 MB came out at 3,989,254 B: −27.7%, which the headline rounds to 28. The two biggest wins are the two formats the dissection redirected, and they are the middle of this post.

The plan lost to plain bytes on our own corpus

The first change looked like the safest bet on the list. Numeric doc-value columns were stored as raw 8-byte values per document; the textbook fix is frame-of-reference plus bit-packing — subtract the minimum, divide out the greatest common divisor, pack the residuals into 128-document blocks at the width the spread demands. That is the shape tantivy's columnar store uses, and our unit tests measure monotone strided columns ~5× smaller than raw before compression.

The first cut shipped bitpack-everywhere. The harness rejected it outright: the .dv family came out at 2.0× the baseline — 789,167 B against 390,487. The optimization doubled the size of the thing it was optimizing.

The reason is the interesting part. The corpus cycles its 4,008 events, so the raw lanes repeat byte-for-byte down the file, and the outer zstd pass matches repeating bytes almost for free. Bit-packing is block-local: every 128-doc block scrambles the byte grid the compressor was matching against. An information-theoretic win lost, on real bytes, to a compressor that could see the whole file.

What shipped instead races both candidates per column — the bit-packed layout and the raw one — compresses both at the real level, and keeps whichever is smaller after zstd: the Parquet pick-the-smallest rule. On this worst-case-for-bitpacking corpus it wins 9.0% on .dv (0.6% of the whole index); on genuinely monotone columns — timestamps, counters — it can win five times as much. The chooser pays both prices and never guesses.

Postings: a percent here, a percent there

The postings files carry, for every term, the sorted document ids and the term frequencies, packed in 128-document blocks. The blocks were paying a framing tax: a 4-byte length that is a pure function of the bit width, and a residual count the reader can derive on its own. ZPS2 re-framed them — [width][vbyte min], where width zero means a constant block that carries no payload at all; the positions block lost its length prefix (exactly 128 count-and-delta groups pin its end); and a positioned term whose total term frequency equals its document frequency omits its entire frequency stream, because the reader can re-derive every count as one.

Measured: .post −3.4% (2,449,405 → 2,364,905 B), body.post −4.3% (1,828,102 → 1,749,695 B). Small and honest — and the widest keyword family in the corpus (top_doc, 12 terms at df ≈ 8.3k) went the other way, +1.5%. Families move in both directions; the family total is what gates the merge.

The stored section, where the dissection redirected the plan

The .seg file holds the stored documents — the columnar copy of _source. The epic's plan for it was dictionary work on keyword columns. Then we dissected one: the planned dict-stream work targeted about 5% of the section. The body text's JSON fallback alone was 84.2% of it. The plan was aiming at the wrong end of the file.

The same dissection showed two cheaper embarrassments: __seq_no — consecutive integers — stored as ASCII text, five bytes for “12345”; and doc_id stored as a byte-identical duplicate of the engine's internal __id column. Three codecs went in instead of the dictionary work:

• TYPED_INT — an all-integer column becomes a presence bitmap plus 128-value frame-of-reference blocks, the same shape as the ZPS2 postings framing, with a zigzag-varint residual for what escapes the blocks. Floats and strings never qualify and keep the JSON fallback.
• COPY_OF — a column byte-identical to an earlier column is written as a 4-byte reference, after a digest plus a full equality check. An id echoed into _source costs 4 bytes instead of a second payload.
• A long-effort chooser, merge path only — each JSON-fallback column is additionally compressed at zstd level 19 with long-distance matching and an explicit 16 MB window; the smaller payload wins. Flush stays pinned to level 3, where an earlier measurement showed level 19 is an ingest regression we had no reason to re-learn.

Two measured reasons the third codec is a chooser and not a level bump. The window knee is sharp: the 29.8 MB body column compresses to 679 kB at the default window and 330 kB at 16 MB, because the corpus's sentence pool repeats just past a small window. And high levels regress small-alphabet streams: the id column is 54 kB at level 3 and 143 kB at level 19. A global 19 would grow the ids while it shrank the body.

Result: .seg −73.1% (1,671,271 → 449,094 B), the whole index −22.6% in one stage (5,398,478 → 4,176,300 B). Every other file family is byte-identical between those two runs but for one byte in a manifest counter — the change is fenced to its own envelope, and anything an older codec could already express is still written in that older codec, byte for byte.

138.7 KB for about 200 bytes of information

The .meta files hold per-term postings metadata: document frequency, total term frequency, and a pointer into the postings — for every term in the field's dictionary. doc_id has 100,000 unique terms, and in this corpus every one of them has ttf == df (each document holds its id once) and every postings gap is zero. The information content of that file is a few hundred bytes. It cost 138,727 B.

The culprit was the record stride: {df u32, ttf u64, offset u64, len u32} — 24 fixed bytes per term, interleaved. zstd did compress that stream, seventeen-fold, but it cannot match across the stride's alternating field widths, so every record still cost it residual bytes on the lanes that vary: about 1.4 bytes per term, 138.7 kB across a hundred thousand terms, for a file whose information content is a few hundred bytes.

The obvious first move — keep the records, delta-encode the fields in place under the envelope — would have kept the stride, and the stride itself was the cost. ZFM5 reshapes the section before compressing it: four columnar varint streams — df, ttf − df (elided entirely when every term in the field has ttf == df, which is the keyword-field shape), offset gaps, and lengths. zstd sees four long runs of one-symbol lanes instead of 100,000 interleaved records. On the final tree that file is 216 B on disk, envelope header included. The reader expands the streams once at open time into the flat record array the previous formats built, so the term-lookup hot path is untouched.

One more thing this format earned before it merged. An adversarial review of the diff — independent reviewers told to break the decoder — confirmed exactly one real defect: the total-frequency reconstruction was the only unchecked addition left in the expansion loop, and a crafted delta could wrap it in a release build to ttf < df, silently flipping the freq-elision predicate from section 03. Unreachable from our writer, but a decoder should not trust its writer. Fixed with checked arithmetic and a regression test that feeds it the crafted delta. .meta finished at −93.1% (200,826 → 13,774 B).

The peer anchor is worth naming: tantivy's term dictionary serializes block-leading absolute ranges once and per-term VInt lengths, reconstructing offsets by accumulation — same insight, MIT-licensed, cited beside our implementation of it. The columnar split and the elision flag are ours.

Six percent of raw — and the caveat that number needs

After the four stages, the index for those 100,000 documents — the whole data directory — is 3,989,254 B. The raw documents are 63,461,256 B of NDJSON, measured off the exact bulk payload the harness indexed. Against the obvious baselines on the same bytes:

The index — which answers queries, accepts updates and deletes, and gives random access into every one of those bytes — lands smaller than gzip -9 of the same text. It will never reach zstd -19 of a sequential stream, and it should not be asked to: that row is a compressor with unbounded CPU and no obligation to ever seek.

Two whales, and a plan for each

Where the 3,989,254 B went, from the committed harness result for the final tree (the .seg family measured 449,094 B in that stage's own A/B — two bytes of run-to-run noise):

Two clusters dominate. Body text is ~2.11 MB, about 53% of the index — postings for the body field (1,749,695 B), its stored column (328,266 B), its length norms (35,194 B). That is 7.1% of the 29.8 MB of raw body text: text is already near its entropy, and the remaining wins there are incremental framing work. The id echo is ~0.88 MB, about 22% — in this corpus _id and doc_id are the same string, and it is stored four ways: the .ids pairs at 613 kB, doc_id's own postings at 256 kB, the __id stored column — 9.4 kB on the merged tree, after stage 1's duplicate-reference and long-effort codecs had their way with it — and term-dictionary slices. That cluster still pays ~45% of its raw bytes, where the body pays 7.1%.

Stage 2 is dictionary reuse: the full-text term dictionary already is a finished dictionary for keyword fields — today we learn the ids and then store them again in two more places; referencing term ordinals from the other columns deletes the echo. Stage 3 is trained zstd dictionaries for the many-small-files tail. Neither is measured yet, so neither gets a number in this post; when they are, the losing rows will be in that one too.

What we tried and kept out

Every number in this post traces to a run. The harness and its design notes are committed in benchmarks/index-size, with the last two stages' result files beside them and the first two stages' A/B tables in the changelog entries; the work is tracked in epic #1038 and landed as pull requests #1040, #1041, #1042 and #1043, each with its own measured table. The wire-compatibility conformance suite stayed green across all four merges — the gate that outranks size.

If you would rather check the claim than read about it: the engine is one binary, the harness is in the repository, and your own corpus is the benchmark that matters. The retrieval side of this engine's story is in our post on reranking and the BM25-vote decisions study; the wire-protocol conformance numbers live on the benchmarks page.