XERJ reads one TOML file. Every key has a production-ready default, so the smallest working config is an empty file. The table below is the full surface; the sections below it walk each group with a runnable example.
Paths for the config are, in precedence order: --config /path/to/xerj.toml on the command line, then /etc/xerj/xerj.toml, then ./xerj.toml in the working directory. If none exist, the full default is used.
Full key table
KEY
TYPE
DEFAULT
DESCRIPTION
[server]
rest_port
u16
8080
Native REST API listener port. All native endpoints (/v1/*) live here.
grpc_port
u16
8081
Reserved for a future gRPC API. Not wired in v0.1 — leave at default.
es_compat_port
u16
9200
Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.
bind_address
string
"127.0.0.1"
Interface to bind every listener to. Loopback by default, so an unconfigured node is unreachable from the network. Set "0.0.0.0" or a private address to expose it (also --bind / XERJ_BIND_ADDRESS). Must be an IPv4 or IPv6 literal — host names are not resolved, and a node given one refuses to start.
allow_insecure_network_bind
bool
false
Permit a non-loopback bind_address while tls.enabled = false. Startup refuses that combination otherwise: every listener would serve plain HTTP, so the API key in every Authorization header crosses the network in cleartext. Set true only when a proxy, sidecar, mesh or container boundary terminates TLS in front of the node. Env: XERJ_ALLOW_INSECURE_NETWORK_BIND.
data_dir
path
"./data"
Root for indices, WAL files, and segments. Needs fast I/O and enough free space.
[auth]
enabled
bool
true
Require an API key on every request. Set to false only on a trusted network.
admin_api_key
string
""
Static admin key. Left empty, a 256-bit key is generated on first run and written to /admin.key.
[tls]
enabled
bool
false
Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.
cert_path
path
""
PEM X.509 certificate. Use a CA-signed cert in production.
key_path
path
""
PEM private key. Permissions should be 0600.
[storage]
wal_sync
enum
"batched"
"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.
wal_batch_ms
u32
100
Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.
Memtable buffer size that triggers a segment flush (MiB).
flush_interval_secs
u32
30
Maximum wall-clock interval between flushes even if the buffer is not full.
[merge]
strategy
enum
"size_tiered"
Only "size_tiered" is implemented; "log_structured" is refused at startup rather than silently running the other policy.
min_segments
u32
10
Accepted and validated (must be ≥ 2) but not wired: the merge trigger is per-tier and comes from min_merge_count. Setting it away from the default logs a warning at startup.
max_segment_mb
u32
8192
Upper bound on a mergeable segment (MiB). Segments at or above this size are never merged again.
io_rate_mb_per_sec
u32
100
Accepted but not wired: merge I/O is not throttled in this build — the rate limiter that honours it sits in an unused storage-crate merge executor. Setting it away from the default logs a warning at startup.
max_concurrent
u8
1
Accepted but not wired: merge parallelism comes from the XERJ_MERGE_PARALLELISM environment variable, which also defaults to 1. Setting it away from the default logs a warning at startup.
[compression]
enabled
bool
true
Accepted but not wired: every durable artifact is a compressed envelope with no uncompressed write path, so false changes nothing on disk. Setting it away from the default logs a warning at startup.
level
enum
"balanced"
"fast" (Zstd L1), "balanced" (Zstd L3, default), "best" (Zstd L6, cold storage). Applied when segments are re-encoded at merge; flush always writes L3, because raising the flush level collapses sustained ingest. Force-merge to apply a change to data already on disk.
block_size_docs
u32
128
Accepted but not wired: the stored codec is columnar over the whole segment section, not blocked by document count. Range 16–4096 is enforced at startup, and a non-default value logs a warning.
[fts]
default_analyzer
enum
"standard"
"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping.
[vector]
default_metric
enum
"cosine"
"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice.
hnsw_m
u32
16
Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.
hnsw_ef_construction
u32
200
Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.
hnsw_ef_search
u32
100
Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).
default_quantization
enum
"none"
"none" (the default) or "scalar8" = kNN scored from 1-byte-per-dimension codes, 1–2% recall loss. scalar8 buys precision, not RAM: the scan reads the full-precision vector from _source and quantizes it per query, so resident memory is unchanged and the field also gives up HNSW-served kNN (#392). "binary" has no quantizer behind it and is refused at startup rather than silently stored at full precision; "scalar4" is not a value this key accepts at all (the 4-bit quantizer in the vector crate is unreachable from config or mapping).
max_dimensions
u32
16384
Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096.
[logs]
retention_days
u32
90
Auto-delete log docs older than N days. 0 disables.
OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".
default_model
string
""
Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".
batch_size
u32
64
Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.
timeout_ms
u32
5000
HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.
[limits]
max_query_memory_mb
u32
512
Per-query memory cap (MiB). Queries that exceed it are cancelled.
max_concurrent_searches
u32
64
Global in-flight search ceiling. Extras are queued.
max_fields_per_index
u32
500
Field-explosion protection. ES default is 1000; 500 is intentionally stricter.
[indexing]
turbo_batch_size
u32
1000
Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency.
turbo_parallel
bool
true
Parallel tokenisation on Rayon threads. Disable only for debugging.
turbo_fast_analyzer
bool
false
Skip stemming and stop-word removal in turbo mode. Trades recall for speed.
[cluster]
enabled
bool
false
Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.
port
u16
9300
TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.
peers
array
[]
Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].
tick_ms
u64
50
Raft tick interval (ms). Lower = faster leader election at the cost of CPU.
auth_secret
string
""
Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.
[wal_tap]
enabled
bool
false
Push a filtered subset of indices to an external ES-compatible target (Elasticsearch, OpenSearch, or another XERJ node) in near-real-time. One-directional and single-node — this is not cross-cluster replication. Off by default.
target_url
string
""
Base URL of the target cluster; the tap POSTs to {target_url}/_bulk. Example: "https://central:9200". Empty disables the tap even with enabled = true. Must not carry credentials in the URL (user:pass@host is refused at startup): target_url is echoed by GET /_xerj/wal_tap, by _stats and in the log. Put the credential in target_auth.
target_auth
string
""
Verbatim Authorization header for the target, e.g. "ApiKey abc123". Write-only: GET /_xerj/wal_tap reports whether one is set, never its value.
indices
array
[]
Index allowlist, glob patterns. Empty ships nothing; ["edge-*","metrics"] ships those. A wildcard never expands to a hidden index, and XERJ's own .xerj* system indices are never shipped whatever this says.
poll_interval_ms
u64
500
How often each allowlisted index's WAL is polled (ms). Range 50–60000. This is the floor on end-to-end latency; the tap adds nothing to the write path.
max_batch_docs
usize
1000
Maximum WAL entries read and shipped per poll, per index. Must be at least 1.
max_batch_bytes
usize
5242880
Maximum _bulk body size (bytes). Bounds one HTTP request against the target's http.max_content_length — a larger batch is split across requests, never truncated.
request_timeout_secs
u64
30
Per-request timeout against the target. A timed-out batch is retried; the cursor does not advance, so nothing is dropped.
max_retry_backoff_secs
u64
60
Ceiling on the exponential, jittered retry backoff when the target is down. Range 1–86400 (one day), enforced at startup as well as by PUT /_xerj/wal_tap.
min_retained_generations
u64
0
Rotated WAL generations kept per shard after every entry in them is durable in a segment, so a tap whose target is briefly unreachable still finds them. At the default 0, WAL retention never waits for the target: an outage longer than storage.flush_interval_secs loses entries and reports them as gaps in GET /_xerj/wal_tap/_stats. This is a bounded floor, not a retention lease — the extra cost is at most n × storage.wal_max_size_mb per WAL shard per index, whether or not a tap is running, so a stalled target can never fill this node's disk. Capped at 64 and range-checked at startup, not only by PUT /_xerj/wal_tap — the config file must not be a way around the bound.
[server]
Network listeners and the data directory. Most deployments only touch data_dir and the bind address.
[server]
rest_port = 8080 # native /v1/* API
es_compat_port = 9200 # ES wire-compatible API
grpc_port = 8081 # reserved
bind_address = "0.0.0.0" # not a default — the default is "127.0.0.1" (loopback only)
allow_insecure_network_bind = true # not a default (false) — required for a non-loopback bind while TLS is off
data_dir = "/var/lib/xerj" # not a default ("./data") — an absolute path is strongly recommended
The default bind is loopback, so an unconfigured node is not reachable from the network. Exposing it while tls.enabled = false refuses to start unless allow_insecure_network_bind says the cleartext exposure is intended — enable TLS and neither line is needed.
[auth]
Static API-key authentication. The first-run admin key is written to <data_dir>/admin.key; subsequent starts reuse it. Clients pass Authorization: ApiKey <key> on every request.
[auth]
enabled = true
admin_api_key = "" # blank → auto-generated on first run
# Or provide your own:
# admin_api_key = "ak_live_c8f9a4…"
[tls]
TLS termination at the server. In Kubernetes or behind a load balancer, leave this off and terminate at the proxy instead — one place to rotate certs, one place to log handshakes.
[tls]
enabled = true # not a default (false) — this block switches TLS on
cert_path = "/etc/xerj/certs/server.crt" # not a default ("") — your certificate
key_path = "/etc/xerj/certs/server.key" # not a default ("") — your private key
[storage]
The WAL and flush tuning. wal_sync is the durability knob everyone looks for — pick "sync" for financial/compliance workloads, "batched" for everything else, and "async" only in benchmarks.
[storage]
wal_sync = "batched"
wal_batch_ms = 100 # fsync every 100 ms
wal_max_size_mb = 1024 # roll WAL every 1 GiB
flush_size_mb = 512 # flush memtable at 512 MiB
flush_interval_secs = 30
[merge]
Segment compaction. size_tiered is the only implemented policy: it merges same-size segments, which is cheap and write-optimal. log_structured is a name in the enum with no merge policy behind it, so the server refuses to start on it rather than silently running size-tiered merging for an operator who picked a levelled policy for its read amplification. If full-range scans are what hurts, that is a real gap in this build — setting the key will not change it.
[merge]
strategy = "size_tiered" # the only one implemented — log_structured is refused at startup
min_segments = 10 # accepted, not wired
max_segment_mb = 8192 # 8 GiB cap
io_rate_mb_per_sec = 100 # accepted, not wired — merges are not throttled
max_concurrent = 1 # accepted, not wired (see XERJ_MERGE_PARALLELISM)
tier_floor_mb = 4
min_merge_count = 4 # the real per-tier merge trigger
max_merge_count = 16 # caps peak merge RAM
[compression]
See Compression for the encoding catalog. This section picks the outer block codec effort only — the inner per-column encodings are chosen automatically at write time.
[compression]
enabled = true # accepted, not wired — no uncompressed write path exists
level = "balanced" # Zstd L1 / L3 / L6, applied at MERGE (flush is always L3)
block_size_docs = 128 # accepted, not wired — 16-4096, range-checked at startup
[fts]
Default analyzer applied to untyped text fields. Override per-field in the mapping when creating an index. See Analyzers for the built-ins.
[fts]
default_analyzer = "standard" # unicode words + lowercase
[vector]
Vector search settings. Unfiltered kNN is served by a persisted HNSW graph with exact rescoring (measured recall@10 1.00 on the official bench query); filtered kNN and SQ8 fields run the exact scan. The hnsw_* keys are accepted and validated but not yet wired: the graph builds with fixed M=16 / ef_construction=200, and the query-time beam width comes from the request's num_candidates (floored at 800). Quantization trades recall for scoring precision, not for memory — see default_quantization below and #392.
[vector]
default_metric = "cosine"
hnsw_m = 16 # accepted for compat; kNN serving is exact
hnsw_ef_construction = 200 # accepted for compat; no effect on results
hnsw_ef_search = 100 # accepted for compat; no effect on results
default_quantization = "none" # "scalar8" = int8 precision, 1–2% recall loss.
# NOT a RAM saving today — see issue #392
max_dimensions = 16384
[logs]
Time-series retention. Log indices are sliced into partitions of time_partition width so retention prunes are O(partitions), not O(documents).
[logs]
retention_days = 30 # not a default (90) — keep 30 days
time_partition = "1h" # 1-hour partitions
[embedding]
Delegates vector generation to an OpenAI-compatible endpoint. Leave default_endpoint empty if clients provide vectors themselves. Token limits are model-specific; the chunker in the ai crate splits long documents to fit the model's window.
[embedding]
# OpenAI:
default_endpoint = "https://api.openai.com/v1/embeddings" # not a default ("" disables auto-embedding)
default_model = "text-embedding-3-small" # not a default ("")
batch_size = 64
timeout_ms = 5000
# Or a local Ollama:
# default_endpoint = "http://localhost:11434/v1/embeddings"
# default_model = "nomic-embed-text"
[limits]
Hard caps to protect the server from runaway queries and mapping explosions. Lower these on shared nodes, raise max_query_memory_mb for aggregation-heavy workloads. Leave max_process_memory_mb out to auto-size the process-wide cap from the machine (8 / 16 / 32 GiB tiers); set an explicit MiB ceiling only to raise headroom on a large host — for example if a big xerj autoindex trips 429 circuit_breaking_exception.
Turbo mode knobs. Turbo is opt-in per-request via POST /v1/indices/:name/turbo-ingest or the X-Turbo: true header on _bulk; these settings only apply when turbo is active.
[indexing]
turbo_batch_size = 2000 # not a default (1000)
turbo_parallel = true
turbo_fast_analyzer = false # true only if recall doesn't matter
[cluster]
Multi-node mode. Default is off — single-node doesn't need a consensus layer. When enabled, the embedded Raft implementation replicates metadata only (schemas, shard assignments, node roster). See Clustering for the full story.
[cluster]
enabled = true # not a default (false) — this block switches cluster mode on
port = 9300 # intra-cluster gRPC + Raft
peers = [ # not a default ([]) — your node roster
"a=10.0.0.11:9300",
"b=10.0.0.12:9300",
"c=10.0.0.13:9300",
]
tick_ms = 50