Troubleshooting

The short list of things that go wrong in production, what they look like, and what to do about them. Every symptom links to the metric, the log line, and the config key you turn.

Symptom: ingest throughput drops after a few hours

Likely cause: merge pressure. Segments are piling up faster than the merger can consolidate them, and the memtable is flushing small segments.

Check:

$ curl -s http://127.0.0.1:8080/v1/metrics | grep -E 'segment_count|merge_duration'
xerj_segment_count{index="logs"} 2847
xerj_merge_duration_seconds_count{index="logs"} 183
xerj_merge_duration_seconds_sum{index="logs"} 1847.2

Fix: raise [storage] flush_size_mb so flushes produce bigger starter segments, and raise [merge] max_merge_count so each pass consumes more of them (watch RSS — peak merge memory scales with it). Set XERJ_MERGE_PARALLELISM above 1 for more concurrent merges on fast NVMe. [merge] max_concurrent and io_rate_mb_per_sec are accepted but not wired in this build, so neither will change anything here.

Symptom: queries time out under load

Likely cause: too many concurrent queries or a single query that grew too large.

Check: look for "query cancelled: max_query_memory_mb exceeded" in the logs or active_searches pegged at max_concurrent_searches.

$ journalctl -u xerj --since "10 min ago" | grep -E 'cancel|timeout|rejected'

Fix: if it's memory — raise [limits] max_query_memory_mb (512 → 1024 or 2048 for aggregation-heavy workloads). If it's concurrency — raise max_concurrent_searches, but check that the host actually has headroom first.

Symptom: node RAM climbs forever

Likely cause: vector fields growing without quantization, or too many indices with large flush_size_mb.

Check:

$ curl -s http://127.0.0.1:8080/v1/metrics | grep memory_usage
xerj_memory_usage_bytes 14200000000  # 14 GB and climbing

Fix: set [vector] default_quantization = "scalar8" (or opt heavy fields in per-mapping) so vectors are stored as 1-byte-per-dimension codes. Or lower flush_size_mb so memtables don't grow unbounded.

Symptom: "WAL replay failed" on restart

Likely cause: the server was killed mid-fsync (power loss, OOM kill). The tail WAL file is torn.

Check: first boot log — "wal replay: truncating torn tail at offset N" is benign (XERJ truncates the torn suffix and continues). A hard "wal replay: checksum mismatch at offset N, refusing to start" is not.

Fix: if truncation worked on its own, nothing to do — the last few seconds of writes are lost but the index is consistent. If it refused to start, run xerj verify --data-dir /var/lib/xerj --repair-wal which truncates the WAL at the last valid entry.

Symptom: "disk full" during a merge

Likely cause: no reservation for merge scratch space. A merge of two N-sized segments needs 2N free until the merge completes.

Fix: lower [merge] max_segment_mb so individual merges stay smaller, or free disk. Once the merge retries and succeeds, old segments are unlinked.

Symptom: cluster flapping — leader changes every few seconds

Likely cause: network latency between peers is high enough that heartbeats miss. Raft responds by calling a new election.

Check:

$ journalctl -u xerj --since "5 min ago" | grep -E 'term|election|leader'
... raft: election timeout, starting new term 17
... raft: received higher term 18, stepping down

Fix: raise [cluster] tick_ms from 50 to 150 or 250 — gives heartbeats more room on a slow network. Never drop the tick interval below the RTT between your worst pair of nodes.

Symptom: search returns stale results

Likely cause: recent docs are still in the memtable and a query on a replica is hitting a node that hasn't replicated them yet.

Fix: pass ?preference=primary on the search query to force routing to the primary shard, or lower [storage] flush_interval_secs so the memtable flushes more often.

Getting help

Collect diagnostics before filing a bug. This command builds a self-contained tarball with the config, recent logs, and a metrics snapshot:

$ xerj support-bundle --out /tmp/xerj-support-$(date +%s).tar.gz
wrote /tmp/xerj-support-1745000000.tar.gz (384 KiB)
contents:
  config.toml
  logs/xerj.log.gz
  metrics.txt
  cluster-health.json
  indices-stats.json

Source · engine/crates/xerj-server/src/main.rs · engine/crates/xerj-common/src/metrics.rs