Compression & encodings

XERJ uses two layers of compression. Column encodings are chosen per-field at write time based on the column's shape. Block compression runs over the encoded column data. The two together give XERJ a smaller on-disk footprint than Elasticsearch — measured 1.61× smaller vs ES 8.13.4 on the reproducible telemetry-corpus scorecard (the exact ratio varies by data shape).

Block compression

Three Zstandard effort levels, controlled by [compression] level:

• fast — Zstd L1. Fastest encode, loosest ratio.
• balanced — Zstd L3. Default.
• best — Zstd L6. Tightest measured ratio per byte of decode window. Use for cold indices you query rarely.

The level applies at merge, not at flush. Freshly flushed segments are always written at L3: flush is the back-pressure-critical bound on ingest, and raising its level once collapsed sustained throughput from 1.55M docs/s to 21k docs/s with 75% of documents rejected. Merges run off that path, and they are what produces the long-lived segments where footprint actually matters. So a level change takes effect as segments merge, not the instant you restart — and re-merging an existing index (POST /{index}/_forcemerge) applies it to data already on disk.

Levels are a write-side choice only. Zstd decoding does not depend on the level a payload was written at, so segments written at different levels coexist in one index with no migration and no reindex.

enabled and block_size_docs are accepted but not wired. Every durable artifact XERJ writes is a compressed envelope with no uncompressed write path, so enabled = false does nothing; and the stored codec is columnar over the whole segment section rather than blocked by document count, so there is no doc-block for block_size_docs (16-4096, default 128) to size. Setting either off its default logs a warning at startup. Both were silently ignored before #318.

Column encodings

Picked automatically at ingest time from this set:

Source · engine/crates/xerj-compress/src/field_codec.rs