ES-compatible API

Port 9200. The compat layer exists so you can point an existing Elasticsearch client at XERJ and keep shipping. It handles the most-used operations. Unsupported calls return a structured not_supported_yet error rather than a 500. What the compatibility promise is tested against — ES version, conformance suite, clients — and the known semantic differences below it are stated in one place: the compatibility boundary on Migrate from ES.

Scroll is a bounded snapshot, not a cursor

Elasticsearch's scroll walks segments as it goes. XERJ's does not: the opening _search?scroll= materialises the whole result set into the scroll context up front and later pages read that snapshot. So the context has a ceiling, and it is the one thing to know before pointing export tooling at it — 10,000 documents. A query whose exact total exceeds it is refused with a 400 illegal_argument_exception instead of being paged and silently truncated. Read a result set of any size with search_after on a unique sort key — that path is unbounded.

This is the common case, not the corner case: helpers.scan() in elasticsearch-py, reindex internals, and most export and backup tooling all scroll a whole index, and whole indices are usually bigger than the cap.

The ceiling above is the one _search?scroll= enforces, on the request's total across every index it resolves. One route is looser: the POST /:index/_search_scroll alias applies the ceiling per index, so a comma-separated multi-index scroll sent there can snapshot the full ceiling from each. That direction is permissive rather than truncating: the documents are in the snapshot and page out (measured: 2×6,000 docs over that route returned 12,000 hits across 120 pages). Note that per-hit _index is not reliable on that route — continuation pages return the raw comma-separated index spec rather than the resolving index, so (_index, _id) is not a distinct key there. Export and reindex tooling should not depend on it. Tracked in #405.

# 11,450 documents in `logs` — over the cap, so the scroll never opens
$ curl -sX POST 'http://localhost:9200/logs/_search?scroll=1m' \
    -H 'Content-Type: application/json' \
    -d '{"size":1000,"query":{"match_all":{}}}'
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Scroll result
set is too large: [11450] matching documents exceed the scroll snapshot window of
[10000]. xerj materialises the full scroll snapshot up front and will
not return a silently truncated result set. Page with [search_after] on a unique
sort key instead, or narrow the query."}], ... },"status":400}

# the unbounded path — sort on a unique key, then feed the last hit's `sort`
# value back as `search_after` until a page comes back empty
$ curl -sX POST http://localhost:9200/logs/_search \
    -H 'Content-Type: application/json' \
    -d '{"size":1000,"sort":[{"_id":"asc"}],"query":{"match_all":{}}}'
{"hits":{"hits":[ ...,
  {"_index":"logs","_id":"10897","_score":null,"_source":{"n":10897},"sort":["10897"]}]}}

# the cursor is that last hit's `sort` array, copied verbatim. Read it out of
# the response; never guess it from the page number. `_id` is a keyword, so it
# sorts LEXICOGRAPHICALLY — "1000" < "999" — and page 1 of numeric-looking ids
# ends at "10897", not "999". Pass ["999"] here and the walk ends after
# 1,010 of the 11,450 documents, with no error — the silent truncation this
# whole page exists to rule out. Sort on a numeric field, or on a keyword
# whose text order is the order you actually want.
$ curl -sX POST http://localhost:9200/logs/_search \
    -H 'Content-Type: application/json' \
    -d '{"size":1000,"sort":[{"_id":"asc"}],"search_after":["10897"],"query":{"match_all":{}}}'
# → 12 pages, all 11,450 documents, no ceiling

The rerank stage — a XERJ extension

This feature sends your data off the machine. rerank is a top-level _search key that Elasticsearch does not have, and the only search-time feature that sends document text off the node (proxy embeddings and the WAL tap also send text off the node when an operator configures them; both are off by default, and docs/RERANK.md lists every outbound connection a node can open). It POSTs the question and the text of the top window hits (default 30, maximum 300) to a third-party relevance judge, and reorders the hits by the 0–1 probability that comes back. That probability replaces _score, so rerank.min_score is an absolute cut-off. It does nothing until the node's operator sets a provider key under [rerank], a request only triggers it by carrying the block, and only fields the response returns are sent.

$ curl -sX POST http://localhost:9200/kb/_search \
    -H 'Content-Type: application/json' \
    -d '{"query":{"match":{"body":"vitamin d bone density"}},"size":5,"rerank":{"min_score":0.5}}'
# → "_rerank":{"applied":true,"score_kind":"probability","judged":30,...} ahead of "hits"
# → a provider that misses its deadline is still a 200: "applied":false, engine order
# → no provider key on the node: 503 rerank_exception, no hits, nothing sent

hits.total and aggregations stay the engine's and describe the full match set, not the window. sort, search_after, collapse, scroll and size: 0 are refused with a 400, and _msearch, search templates, _async_search and _rank_eval (per request, under failures) refuse the block by name rather than dropping it. The strings have server-side ceilings like the window does: rerank.instructions is limited to 2,000 characters because the provider's wire format repeats it once per judged document, the question to 4,000 and model to 128; a longer value is a 400 naming the field. Every page request judges the window again, so fetch the window once and page client-side when cost matters. GET /_xerj/rerank (superuser only) reports whether a provider is configured, never the key. Ranking quality with the provider's model has not been verified by this project. Full reference: docs/RERANK.md · the walkthrough.

Bulk indexing example

$ curl -sX POST http://localhost:9200/logs/_bulk \
    -H 'Content-Type: application/x-ndjson' \
    --data-binary '@-' <<'JSON'
{"index":{"_id":"1"}}
{"@timestamp":"2026-04-14T12:00:00Z","service":"auth","level":"error","message":"login failed"}
{"index":{"_id":"2"}}
{"@timestamp":"2026-04-14T12:00:01Z","service":"auth","level":"warn","message":"rate limited"}
JSON

Source · engine/crates/xerj-api/src/es_compat.rs · engine/crates/xerj-api/src/rerank_stage.rs