03 · START

Migrate from Elasticsearch

XERJ runs a second API listener on 9200 that speaks the Elasticsearch wire format. The fast path to trying XERJ is to point your existing ES client at it and let the compat layer take the calls. It isn't 100% coverage; this page says exactly what is and isn't there.

What we test against

The wire target is Elasticsearch 8.13 REST. The compat listener advertises 8.13.0 at GET / and sends X-Elastic-Product: Elasticsearch on its responses — the two handshake gates Kibana and the official clients check before they will talk to a node. Suite cases gated on 8.14+ version predicates are not run; nothing above 8.13 is claimed.
The ES REST API YAML conformance suite, on every commit, gated at zero failures. A full run on 2026-09-20 passes 1,371 / 1,374 cases (3 skipped; the count grows as cases are added — the invariant the gate enforces is 0 failed, not a pass total). The suite is a curated 202-file subset of Elasticsearch's public rest-api-spec YAML tests, extended with XERJ-authored cases for the AI-native surface. Two limits stated plainly: the runner does not verify error behaviour (catch: expectations are unasserted), and it speaks raw HTTP — no ES client library is exercised in CI.
A live Elasticsearch 8.13.4, head to head. The published benchmarks drive one keep-alive HTTP client, the same corpus and the same machine against both engines, and publish every cell including the losses — the matrix is public.
Clients, honestly. There is no Kibana end-to-end test and no client version matrix in this repo. What is verified is the wire itself: the YAML suite above, plus curl-shaped request recipes that run end to end. Verify the specific behaviour your client depends on against your own node before you cut over.

What works day one

Index CRUD: PUT /:index, DELETE /:index, GET /:index, PUT /:index/_mapping, GET /:index/_mapping.
Document CRUD: POST /:index/_doc, PUT /:index/_doc/:id, GET /:index/_doc/:id, DELETE /:index/_doc/:id.
Bulk: POST /:index/_bulk with index/create/update/delete actions.
Query DSL: 50 query types including match, bool, range, wildcard, phrase, span, fuzzy, and AI-native (knn, semantic, hybrid) — the full list is on Query types. A further 2 are recognised and deliberately refused with a 400 (has_child, has_parent): XERJ never materialises a parent/child join, so running them would match against flat documents and return silently wrong hits.
Aggregations: 62 aggregations across the metric, bucket, sampling, scope, geo and pipeline families — bucket_script, derivative, moving_fn and the rest of the pipeline family included. Full list on Aggregations.
Cluster endpoints: GET /_cluster/health, GET /_cat/indices.
Delete by query: POST /:index/_delete_by_query.

What needs rewriting

Whole-index scroll reads. 10,000 documents is the ceiling on a _search?scroll= snapshot: XERJ materialises the snapshot up front, so a query whose exact total is larger is refused with a 400 rather than paged and silently truncated. Anything that scrolls an entire index — helpers.scan(), exports, reindex scripts — moves to search_after on a unique sort key, which is unbounded. Mind that _id sorts lexicographically; take each cursor from the previous page's last sort value rather than computing it. Details, and the one route that applies the ceiling per index instead, on the ES-compat API page.
Scripted fields / scripted metrics. Painless is not implemented. Use function_score for the cases it covers.
Runtime fields. Not supported. Compute at ingest or query time.
Cross-cluster search. Single-cluster only in v0.1.
Machine-learning jobs. Not a feature.
Watcher / alerting framework. Use the /v1/explain-plan endpoint + an external rule runner.
ILM (Index Lifecycle Management). Retention is controlled per-index via [logs] retention_days.

Known semantic differences

What follows are differences on APIs that are implemented: the call succeeds and returns the ES response shape, but the bytes are XERJ's own. They bite when a client asserts on details rather than shapes.

Scroll is a bounded snapshot, not a segment-walking cursor. The headline difference, because it can silently change results for tooling that paginates a live index: _search?scroll= materialises the whole snapshot up front, and a request whose exact total exceeds the window is refused with a 400 rather than paged past it. The ceiling, the per-index alias quirk, and the unbounded search_after replacement are in What needs rewriting above and on ES-compat API.
BM25 scores are XERJ's own. Ranking order matches the query's intent, but exact _score floats differ from ES and can shift for the same query as background merges move collection statistics — the same class of internal difference as between two ES minor versions. Relax tests that assert on exact score values.
The default _search source is compact. With _source omitted, ordinary source fields come back but engine-generated <field>_vector and <field>_vector_chunks companions do not. Explicit _source: true returns them.
_cat headers. _cat/indices?v emits data rows only — no column-header row — though the column order is ES's, so positional parsing keeps working and header-line parsing does not. The per-index form _cat/indices/:index returns that index's row; an unknown index name is the 404.

What's different (for the better)

No heap tuning. No -Xmx, no GC pauses, no page-fault storms.
Single binary. 11 MB static executable vs 620 MB ES tarball.
First-class explain plan. /v1/explain-plan returns the optimizer tree, not a profile dump.
Columnar ingest. Encodings chosen at write time, per column — delta-of-delta, dictionary, ZSTD, SQ8 for vectors.
Vector dims up to 16384. ES caps at 4096.

A 10-minute migration

# 1. run XERJ alongside ES
$ xerj --config xerj.toml &

# 2. reindex one index, one-shot
$ curl -sX POST http://es:9200/logs-2026-04/_search?scroll=1m \
    -d '{"size":1000,"query":{"match_all":{}}}' \
    | jq -c '.hits.hits[]._source' \
    | xerj-ingest http://localhost:9200 logs-2026-04

# 3. compare one query
$ diff <(curl -s http://es:9200/logs-2026-04/_search -d @q.json) \
       <(curl -s http://localhost:9200/logs-2026-04/_search -d @q.json)

Source · engine/crates/xerj-api/src/es_compat.rs

◀ PREVInstall

NEXT ▶CLI