ES-compatible API

Port 9200. The compat layer exists so you can point an existing Elasticsearch client at XERJ and keep shipping. It handles the most-used operations. Unsupported calls return a structured not_supported_yet error rather than a 500.

Scroll is a bounded snapshot, not a cursor

Elasticsearch's scroll walks segments as it goes. XERJ's does not: the opening _search?scroll= materialises the whole result set into the scroll context up front and later pages read that snapshot. So the context has a ceiling, and it is the one thing to know before pointing export tooling at it — 10,000 documents. A query whose exact total exceeds it is refused with a 400 illegal_argument_exception instead of being paged and silently truncated. Read a result set of any size with search_after on a unique sort key — that path is unbounded.

This is the common case, not the corner case: helpers.scan() in elasticsearch-py, reindex internals, and most export and backup tooling all scroll a whole index, and whole indices are usually bigger than the cap.

The ceiling above is the one _search?scroll= enforces, on the request's total across every index it resolves. One route is looser: the POST /:index/_search_scroll alias applies the ceiling per index, so a comma-separated multi-index scroll sent there can snapshot the full ceiling from each. That direction is permissive rather than truncating: the documents are in the snapshot and page out (measured: 2×6,000 docs over that route returned 12,000 hits across 120 pages). Note that per-hit _index is not reliable on that route — continuation pages return the raw comma-separated index spec rather than the resolving index, so (_index, _id) is not a distinct key there. Export and reindex tooling should not depend on it. Tracked in #405.

# 11,450 documents in `logs` — over the cap, so the scroll never opens
$ curl -sX POST 'http://localhost:9200/logs/_search?scroll=1m' \
    -H 'Content-Type: application/json' \
    -d '{"size":1000,"query":{"match_all":{}}}'
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Scroll result
set is too large: [11450] matching documents exceed the scroll snapshot window of
[10000]. xerj materialises the full scroll snapshot up front and will
not return a silently truncated result set. Page with [search_after] on a unique
sort key instead, or narrow the query."}], ... },"status":400}

# the unbounded path — sort on a unique key, then feed the last hit's `sort`
# value back as `search_after` until a page comes back empty
$ curl -sX POST http://localhost:9200/logs/_search \
    -H 'Content-Type: application/json' \
    -d '{"size":1000,"sort":[{"_id":"asc"}],"query":{"match_all":{}}}'
{"hits":{"hits":[ ...,
  {"_index":"logs","_id":"10897","_score":null,"_source":{"n":10897},"sort":["10897"]}]}}

# the cursor is that last hit's `sort` array, copied verbatim. Read it out of
# the response; never guess it from the page number. `_id` is a keyword, so it
# sorts LEXICOGRAPHICALLY — "1000" < "999" — and page 1 of numeric-looking ids
# ends at "10897", not "999". Pass ["999"] here and the walk ends after
# 1,010 of the 11,450 documents, with no error — the silent truncation this
# whole page exists to rule out. Sort on a numeric field, or on a keyword
# whose text order is the order you actually want.
$ curl -sX POST http://localhost:9200/logs/_search \
    -H 'Content-Type: application/json' \
    -d '{"size":1000,"sort":[{"_id":"asc"}],"search_after":["10897"],"query":{"match_all":{}}}'
# → 12 pages, all 11,450 documents, no ceiling

Bulk indexing example

$ curl -sX POST http://localhost:9200/logs/_bulk \
    -H 'Content-Type: application/x-ndjson' \
    --data-binary '@-' <<'JSON'
{"index":{"_id":"1"}}
{"@timestamp":"2026-04-14T12:00:00Z","service":"auth","level":"error","message":"login failed"}
{"index":{"_id":"2"}}
{"@timestamp":"2026-04-14T12:00:01Z","service":"auth","level":"warn","message":"rate limited"}
JSON

Source · engine/crates/xerj-api/src/es_compat.rs