How do I rerank search results with a calibrated relevance judge?
TL;DR — Add "rerank": {} to a XERJ _search body. The top 30 hits go to an external judge, which returns a 0 to 1 relevance probability per document. That probability replaces _score, so min_score becomes a real cut-off. It sends document text off the machine. XERJ has not verified ranking quality with the real model.
Act as a coding agent. Read https://xerj.org/llms.txt, ask the operator whether a rerank provider key is set on the XERJ node and whether document text may leave the machine, then run one search without the rerank block and one with it, and report the _rerank block of the second response, including whether applied is true.
curl -s 'http://127.0.0.1:9200/_xerj/rerank'
Ask the node whether a rerank provider is configured. The answer names where the key came from and never prints the key. A node started with --insecure needs no credentials. On any other node the route is superuser-only, so send the admin key from the admin.key file in the data directory.
curl -s -XPOST 'http://127.0.0.1:9200/kb/_search' -H 'content-type: application/json' -d '{"query":{"match":{"body":"vitamin d bone density"}},"size":5,"rerank":{"min_score":0.5}}'
Rerank the top 30 hits and keep the ones the judge scores at 0.5 or higher. This request sends the text of those hits to the provider.
curl -s -XPOST 'http://127.0.0.1:9200/kb/_search' -H 'content-type: application/json' -d '{"query":{"match":{"body":"vitamin d bone density"}},"_source":["title"],"rerank":{}}'
Send titles only. The judge sees exactly what the response returns, so source filtering is also the control on what leaves the machine.
What the rerank stage does
The engine retrieves and ranks as usual. The stage then sends the question and the top window hits to a relevance judge. The judge answers one yes-or-no question per document and returns a probability that the document is relevant.
XERJ reorders the hits by that probability. The probability replaces _score, and hits.max_score becomes the top hit's probability.
{
"query": { "match": { "body": "vitamin d bone density" } },
"size": 5,
"rerank": { "min_score": 0.5 }
}
The provider is TypeSafe AI's Jev model, reached through its System One API. XERJ asks one question per document rather than one multiple-choice question across all of them, because a real corpus has more than one relevant document. That request shape is the approach used by the hev/jev-rerank project, which is Apache-2.0.
Why a probability is worth more than a better order
A BM25 score orders results but has no absolute meaning. A score of 7.2 on one query is not comparable to 7.2 on another query, so you cannot use it as a threshold.
A calibrated probability means the same thing on every query. rerank.min_score: 0.5 drops every document the judge scored below 0.5, whatever the query was. The top-level min_score stays the engine's threshold on engine scores, and the two compose.
This sends your data off the machine
Reranking is the only search-time feature that sends document text to a third party. Two other features send text off the node, and both are off by default. Proxy embeddings ([embedding] default_endpoint) send document text at indexing time and query text at search time to an external embeddings API. The WAL tap replays every write on tapped indices to an external _bulk endpoint. An operator has to turn either one on.
The node has two more outbound connections, and neither carries document or query text. Neural mode downloads its model from the HuggingFace Hub once. Cluster mode, which a default single-node deployment never starts, sends Raft messages, such as index names and mappings, to your own peers. The reference, docs/RERANK.md, lists every outbound connection a node can open.
A rerank request POSTs the question and the text of up to window hits to the provider.
Three controls exist. Reranking does nothing until an operator sets a provider key. A search only triggers it by carrying a rerank block. An operator can forbid it with enabled = false under [rerank], which refuses every rerank request with HTTP 403.
The judge sees exactly what the response returns, and nothing else. Source filtering is therefore also the control on what leaves the machine: "_source": ["title"] sends titles only. rerank.fields is exhaustive, so ["body"] sends the body and not even the title. Document ids, index names, numbers and vectors are never sent.
Set the provider key
Put the key in the config file or in the environment. A value in the config file wins over the environment. A stray variable therefore cannot silently change where document text is sent.
[rerank]
enabled = true
api_key = "ts-..."
endpoint = "https://api.typesafe.ai/v1/systemone"
The environment fallbacks are TYPESAFE_API_KEY and TYPESAFE_ENDPOINT. There is no per-request key, because a key in a search body would land in logs. GET /_xerj/rerank reports whether a key is set and where it came from. It never returns the key.
Read the _rerank block
Every reranked response carries a _rerank block ahead of hits. Read applied before you trust the order.
| Field | Meaning |
|---|---|
applied | true means the order is the judge's. false means the engine's order stands, and reason says why. |
score_kind | probability or engine. |
judged | Documents the provider returned a verdict for. |
pruned_below_min_score | Judged hits removed by rerank.min_score. |
fields_without_text | Named fields that no hit in the window returned text for. |
usage | Tokens the provider reported for this search. |
hits.total and aggregations stay the engine's. They describe the full match set, not the window. When four documents match and two are judged irrelevant, total is 4 and pruned_below_min_score is 2.
What happens when the provider fails
The policy is split: degrade on deadline, surface on contract.
| What happened | Result |
|---|---|
The provider missed timeout_ms | HTTP 200, engine order, applied: false. |
The rerank block is malformed or the combination is refused | HTTP 400, nothing sent. |
| The operator disabled reranking | HTTP 403, nothing sent. |
| No provider key on the node | HTTP 503, nothing sent. |
| The provider answered 401 or a malformed body | HTTP 502, no hits. |
A slow third party is not a reason to deny you results you already have. A wrong key does not fix itself, and hiding it would mislead you about which ranking you hold. This split follows the approach Meilisearch uses in its personalization module (MIT). The approach and the retry back-off constants were adapted from it, and the code cites the lines.
Cost facts
Every document in the window is a paid judgement. The window defaults to 30 and the server caps it at 300. A provider call carries at most 30 documents, so window: 35 is two calls. Scores from different calls stay comparable, because each one is an absolute probability rather than a rank inside its batch.
XERJ sends 8 calls at once by default and 16 at most. The usage field is the provider's own token count for the search. XERJ does not price it.
Every request is judged from scratch. There is no verdict cache, so three page requests over one 30-document window are three provider calls and 90 paid judgements. To keep the cost down, fetch the window once and cut the pages out of it in your own code. Page two continues page one only if the provider returns the same probabilities on a repeat call, which XERJ has not verified for the real model.
The strings have ceilings too. instructions is limited to 2,000 characters, because the provider's wire format repeats it once per judged document. The question is limited to 4,000 characters and model to 128. A longer value is an HTTP 400 that names the field and the limit.
What is not verified
XERJ had no provider key. Ranking quality with the real Jev model is not verified by this project. Every test of the stage runs against a test double that scores by a table the test supplies.
The hev/jev-rerank README publishes nDCG@10 of 0.768 on SciFact and 0.358 on NFCorpus for Jev. XERJ did not run those systems, and they rerank that project's own first-stage shortlist, not XERJ's. XERJ's own hybrid search scored 0.6993 and 0.3448 on the same datasets. Those two figures were measured with --embed-mode neural and the all-MiniLM-L6-v2 model. The default embedder is lexical feature hashing, which has no model in it. This is not a controlled comparison.
When reranking is not worth it
Do not rerank an exact identifier lookup, a filter, or an aggregation-only request. The first stage already answers those, and the stage adds a paid network call per search. Rerank when the right document is probably in the top 30 but not at the top, or when you need an absolute relevance cut-off.
FAQ
How do I rerank search results in XERJ?
Add a rerank object to the body of POST /{index}/_search. An empty object uses every default: the top 30 hits are judged and reordered by relevance probability.
Does reranking send my documents to a third party?
Yes. It is the only search-time feature that sends document text off the node. Proxy embeddings and the WAL tap also send text off the node when an operator configures them, and all three are off by default. The question and the text of up to window hits go to the provider. Only fields the response returns are sent.
What happens when the rerank provider is slow?
The search still answers with HTTP 200. You get the engine's order and scores, and the _rerank block says applied: false with the reason. A slow provider degrades; it does not fail the search.
What happens when the provider key is wrong or missing?
The search fails loudly. A missing key is HTTP 503, a provider 401 is HTTP 502, both with type rerank_exception and no hits. XERJ does not hand back lexical order as if it had been reranked.
Is the ranking quality of the rerank model verified?
No. XERJ had no provider key, so ranking quality with the real model is not verified by this project. The tests use a test double. Only the mechanism is verified.
Can I combine rerank with sort or aggregations?
Aggregations work and still describe the full match set. sort, search_after, collapse, scroll and size: 0 are refused with an HTTP 400 that names the conflict.
Can an AI agent ask for reranking through MCP?
Yes. The xerj_search and xerj_hybrid_search tools take an optional rerank argument. It only works when the operator has set a provider key on the node.
Evidence
- Reranking is the only search-time feature that sends document text off the node. Proxy embeddings and the WAL tap also send text off the node when configured; the neural model download and cluster Raft messages carry no document or query text. A test checks that list against every outbound network client in the engine source. —
engine/crates/xerj-rerank/tests/egress_inventory.rs - The rerank stage defaults to a window of 30 hits, allows at most 300, sends at most 30 documents per provider call, runs 8 calls in flight by default and 16 at most, and defaults to a 10000 ms stage budget capped at 60000 ms. —
engine/crates/xerj-rerank/src/lib.rs - The failure policy, every refused combination, the 400, 403, 502 and 503 statuses, paging inside the window, and the rule that only returned text is sent are each pinned by an HTTP test against an in-process stub of the provider wire format. —
engine/crates/xerj-api/tests/rerank_stage_http.rs - The nDCG@10 figures for Jev (0.768 SciFact, 0.358 NFCorpus), Voyage rerank-3 (0.755, 0.357) and Cohere rerank-v3.5 (0.745, 0.340) are published in the hev/jev-rerank README. XERJ did not run them, and they rerank that project's own first-stage shortlist. — https://github.com/hev/jev-rerank
- XERJ hybrid RRF scored nDCG@10 0.6993 on SciFact and 0.3448 on NFCorpus with --embed-mode neural and all-MiniLM-L6-v2, not with the default lexical embedder. —
benchmarks/beir-hybrid/README.md