Storage & WAL

XERJ writes one WAL per index and a list of immutable segments. Each segment is three files, not a segment directory with twelve — a data file, a skip index, and a doc-id sidecar. All three are mmap'd, so reads come straight out of the OS page cache with no application-side buffer.

data/
├── logs/                      · an index
│   ├── schema.json            · field mapping
│   ├── wal/
│   │   ├── wal-000001         · append-only
│   │   └── wal-000002         · rolls at wal_max_size_mb
│   ├── seg-000001/
│   │   ├── segment.seg        · columnar data, mmap'd
│   │   ├── segment.sidx       · skip index for seeks
│   │   └── segment.ids        · doc-id sidecar (external id ↔ internal ordinal)
│   └── seg-000002/
│       ├── segment.seg
│       ├── segment.sidx
│       └── segment.ids
├── traces/
│   └── ...
└── cluster/                   · Raft metadata, only present in clustered mode
    ├── raft-log-*
    └── snapshots/

WAL

Append-only per index. Generation-rotated at wal_max_size_mb (default 1024 MiB). Retained until the flush checkpoint passes the tail generation, then the old file is released. Fsync policy is controlled by [storage] wal_sync:

• sync — fsync before acknowledging each write (one fsync per _bulk request = group commit). Power-loss durable, matching Elasticsearch's default index.translog.durability: request. Lowest throughput; financial or compliance workloads.
• batched (default) — writes reach the kernel immediately, so an XERJ process crash loses nothing; a background loop fsyncs every wal_batch_ms (default 100 ms), so a power loss or kernel panic can lose at most the last ~100 ms of acknowledged writes. Deliberately below Elasticsearch's request-fsync default in exchange for throughput — set wal_sync = "sync" when you need power-loss durability.
• async — never fsync; the OS decides when to flush. Maximum throughput; a power loss or OS crash can lose up to ~1 s. Dev/bench only.

Segments

Three files per segment:

• segment.seg — columnar data. Each field is a separate column run; queries only touch the columns they actually read. Integer columns use delta-of-delta or FOR; string columns use dictionary + FST. The whole file is mmap'd read-only at open time.
• segment.sidx — skip index. A sparse per-column seek table so point-lookups don't scan the whole column. Mmap'd alongside the segment.
• segment.ids — doc-id sidecar mapping external document ids to internal ordinals. Kept outside the main file so id lookups don't pull columnar pages into memory. Mmap'd read-only.

Segments are immutable once written. Updates and deletes work by writing a new segment and a tombstone; merges rewrite surviving documents into a larger segment.

Merges

[merge] max_segment_mb excludes anything that big from further merging (default 8192). tier_floor_mb (default 4) sets the tier boundaries, min_merge_count (default 4) is the per-tier trigger, and max_merge_count (default 16) caps how many segments one batch merges — and so caps peak merge RAM. Those four are what the merge path reads.

strategy has exactly one implemented value: size_tiered. log_structured is refused at startup rather than silently substituted, so a config that asks for it fails loudly instead of running the other policy.

Three further keys in this section are accepted but not wired, and setting any of them away from its default logs a warning at startup: min_segments (superseded by min_merge_count), io_rate_mb_per_sec — merge I/O is not throttled in this build; the rate limiter that honours it sits in a storage-crate merge executor the engine never constructs — and max_concurrent, since merge parallelism comes from the XERJ_MERGE_PARALLELISM environment variable (default 1).

Cluster metadata

In single-node mode there is no cluster/ directory — everything the engine needs is right next to the index data. When the server is started with a cluster config, a sibling cluster/ directory holds the embedded Raft log and snapshots. Index data is never in Raft — only the metadata (index schemas, shard assignments, node roster). See Clustering.

Source · engine/crates/xerj-storage/src/segment.rs · engine/crates/xerj-storage/src/lib.rs