DOES XERJ BEAT JEV?
ON THE BILL, OUTRIGHT. ON FIQA, NO.

FIRST, WE BROKE IT
IN PUBLIC.

The rerank stage takes BM25's top hits, sends them to a hosted relevance judge, and reorders by the probability that comes back. It shipped with a caveat that said: quality not verified, no provider key. Then TypeSafe gave this project an evaluation key. The first measured run put a number on the caveat. The number was a disaster. The judge made ranking worse than not using it.

• The lesson travels. A judge model answers the question you actually sent, token by token. Your code can score well and still ask a different question than the one you think you asked. Measure the integration at the raw wire before you trust any number that comes back.

Raw artifacts: benchmarks/beir-hybrid/results/2026-09-20-rerank-pilot and …/2026-09-20-rerank-full · narrative in docs/research/typesafe-skill-2026-09/NOTES.md · model pinned jev-1.13.0 throughout.

THEN THE WHOLE
DATASETS. THREE TIMES.

Three BEIR datasets, all public, all from the standard distribution. SciFact has 300 judged queries. NFCorpus has 323. FiQA has 648. We fetched BM25 top-30 shortlists once. The shortlist step is deterministic. Then we sent the fixed request shape three times per dataset, with the query order shuffled each time, and took per-query medians. Zero failed requests. We repeated because the provider is not deterministic. That finding is below, and it sets the resolution of everything in this section.

Before calling any model we checked our harness on FiQA. Our local BM25 scored 0.2382 nDCG@10. BEIR's published BM25 figure for FiQA is 0.236. Close enough to trust the scorer.

• Read the Run-by column before the numbers. The three hosted rows rerank that project's own first-stage shortlists, not ours. Same dataset, same metric, nothing else controlled. Our Jev rows rerank XERJ's own BM25 top-30. That is the setup an operator can actually run. The published 0.768 and our 0.7410 are not a race. They are two different experiments that share a metric.
• Where the judge wins big, and where it does not. FiQA is the biggest lift, +0.126. Finance questions live on phrases like "business expense on a business trip", and BM25's word overlap buries good documents deep. The judge reads them and pulls them up. On SciFact it beats our best local arm too, 0.6993 hybrid to 0.7410 reranked. On NFCorpus it loses to the local hybrid, 0.3312 against 0.3448. The losses sit in medical shorthand, where 30-candidate windows are mostly noise to a general judge, and in the 25 queries where BM25 returned nothing at all. No reorder can fix an empty list.

• About the calibration number you will meet again below. The local history vote on this site measured ECE 0.012 on Banking77. That is not a scoreboard win over the judge's 0.10 to 0.31. The vote did in-domain classification against its own training labels. The judge was scored cross-domain against graded human relevance labels. Calibration belongs to the task, not to the interface. Both numbers are honest on their own task. Neither redeems the other.

Runner and raw JSON: …/2026-09-20-rerank-full (SciFact, NFCorpus) and …/2026-09-20-rerank-full-fiqa (FiQA, with dataset provenance) · hybrid-row figures (0.6993 / 0.3448, 2026-09-18, rc.74): benchmarks/beir-hybrid (results/scifact.txt, results/nfcorpus.txt) · run 2026-09-20 on the wire-compat branch engine, scoring byte-identical to the hybrid benchmark's eval.py · FiQA fetched from the standard BEIR distribution · hosted-reranker figures: hev/jev-rerank (MIT). A third party published them. Not TypeSafe's numbers, and not ours.

THEN WE ASKED THE
QUESTION BACKWARDS.

System One is a documented interface for typed judgement. You POST a state, a model, and a map of questions. Each question comes back noul, a yes/no with a 0 to 1 probability, or choice, N named options with probabilities. XERJ already had a fast answer for exactly that shape of question. Index the labelled history you already have. Answer a new question by a rank-weighted vote over its k nearest neighbours. Report the winning label's vote share as the probability. No model. No tokens. So the node learned to speak the wire.

The gate: the pip client, unmodified, on a local node

No SDK of ours, no patches, no subclassing. pip install jev-reranker (0.1.0, MIT) in a clean venv, its own listwise template, its own splitting, its own strict response validation, pointed at a XERJ node holding 4,000 labelled SMS messages. Two environment variables, and one of them is a trap:

$ pip install jev-reranker
$ # TYPESAFE_ENDPOINT carries the FULL path. TYPESAFE_BASE_URL is silently ignored
$ # by this client (reranker.py:238), and the call still goes to the real host.
$ export TYPESAFE_ENDPOINT=http://localhost:9441/v1/systemone
$ export TYPESAFE_API_KEY=local-only        # the node ignores it; localhost only
$ python3 gate_run.py
client resolved model: ['xerj-history-vote-1']
  1.0000  spam  Please call our customer service representative on 0800 169 6031 betwe
  1.0000  spam  Great News! Call FREEFONE 08006344447 to claim your guaranteed £1000 C
  1.0000  spam  URGENT! Your Mobile number has been awarded with a £2000 prize GUARANT
  0.0000  ham   If you're not in my car in an hour and a half I'm going apeshit
  0.0000  ham   Carlos says we can pick up from him later so yeah we're set
  ...                                                     # 10 fixed docs, 5+5
mean noul  spam=0.9014  ham=0.0956  gap=0.8058
GATE PASSED: unmodified jev-reranker ranked from the XERJ node

The client's own validation is the wire gate. Answers keyed by the sent ids. Every noul numeric in 0..1. Model a non-empty string. Usage non-negative integers. Surviving it unmodified is the compatibility claim. The spam/ham separation is the semantic one. Zero requests left the machine. The vote is an ordinary search of an ordinary index, and the module adds no outbound client.

Speed: the Rust side of the same wire

A full news item against 120,000 labelled AG News examples is a heavy query. About 40 terms, and the match set is most of the index. This is the workload that exposed the waste in our own search loop. The old path built a hit list per term, folded them through hash maps, and sorted the whole match set. All that to return a size-10 page. We replaced it with a WAND-style pruned walk for the one shape every match query becomes. A document is now scored only when it can still make the page even at its best possible score. The method is the classic one from Broder et al., and the same idea tantivy uses, adapted to our postings format.

2.0× faster end to end, and the 7,600 decisions are bit-identical: same labels, same confidence bands. 83.1% of items decided at confidence 0.8 or higher, and 96.7% of those right. Zero items had no neighbour. Ranking quality did not move because scores are computed in the same order and to the same values. We proved that three ways. The three BEIR runs reproduce to four decimals on the new engine, 0.6572, 0.3016, 0.2382. A 648-query A/B over a byte-identical copy of the FiQA index found zero order changes and zero score-bit changes. The full ES wire-conformance suite on that branch stayed at 1371 passed, 0 failed — a dated capture from the wire-compat branch these runs measured; the site-wide canonical figure on every commit is on the benchmarks board.

What the vote itself is worth

Measured earlier on the same engine (the hub's ranking section), on the vote's home ground: labelled history, in-domain classification. No judge model was run on these datasets by us, and none of these rows say anything about Jev.

$ # the audit surface — same vote, evidence attached
$ curl -s localhost:9440/_decide -H 'content-type: application/json' \
    -d '{"index":"sms","question":"URGENT! Your Mobile number has been awarded"}'
{"index":"sms","k":10,"label":"spam","confidence":1.0,"abstain":false,
 "neighbours":[{"_id":"1805","label":"spam",...,"weight":1.0}, ...]}

• Read this before quoting the compatibility claim. The node did not "become a language model." It answers a typed judgement wire from a search over your own labelled history. It says so in its model field. It refuses what it cannot know: zero support, score-type ordinal questions. It attaches the neighbours behind every answer. If you need judgement where you have no history, a new policy, an unseen category, this is the wrong tool and the hosted judge is the right one. The two ends of the wire are complements, not competitors.
• Latency scales with the match set, not with the corpus. AG News items are long and match nearly everything, so they cost tens of milliseconds. Short texts on the same engine cost far less: SMS is 0.9 ms per message. One shape at one scale, measured. No TB-scale claim.

WAND engine change: cf6a9989, with the A/B logs · AG News artifacts and provenance: benchmarks/decisions-as-retrieval/results/2026-09-20-agnews · endpoint: engine/crates/xerj-api/src/systemone_api.rs · 14 end-to-end tests: tests/systemone_http.rs · operator doc: docs/DECISIONS.md · gate harness and transcript: benchmarks/systemone-gate.

JUDGEMENT IS RENTED.
DECISIONS ARE OWNED.

The judge's bill on the 300-query SciFact run, next to the vote's numbers on its own measured workloads. Two tools, two workloads, priced side by side:

• Both columns survive contact with the other's weakness. The judge is the only option where there is no history. And every call is metered, slower, and ships document text to a third party. The vote is free, tens of milliseconds, and never leaves the node. And it cannot judge anything your history does not already cover. At 300 queries a day the judge costs $0.40 and is irrelevant to your budget. Use it freely. At 300,000 messages a day, at this run's measured 1.49 s per call, that is over 120 hours of provider latency per day, however you parallelise it. It is roughly $400 a day at this run's measured rate. And every one of those messages leaves the node. The vote answers each in tens of milliseconds, on the node, for nothing.
• Why the wire matters more than either number. Every System One client, the reranker, the SDKs, whatever an agent is already holding, now has a second local implementation of the interface it speaks. Point TYPESAFE_ENDPOINT at a XERJ node when the questions have history and must stay home. Point it at the hosted API when they do not. Same client. Same wire. No code change. That is the whole bridge.

Reproduce the gate

$ git clone https://github.com/xerj-org/xerj && cd xerj/engine
$ cargo build --release -p xerj-server
$ cat > gate.toml <<'EOF'
[decisions]
index          = "sms"
k              = 10
label_field    = "label"
text_field     = "text"
positive_label = "spam"
min_confidence = 0.0
EOF
$ ./target/release/xerj --insecure --port 9440 --config gate.toml # native REST on 9441
$ python3 benchmarks/systemone-gate/gate_load_sms.py   # index 4,000 labelled SMS
$ python3 -m venv ~/gate-venv && ~/gate-venv/bin/pip install jev-reranker
$ TYPESAFE_ENDPOINT=http://localhost:9441/v1/systemone \
  TYPESAFE_API_KEY=local-only ~/gate-venv/bin/python benchmarks/systemone-gate/gate_run.py
mean noul  spam=0.9014  ham=0.0956  gap=0.8058
GATE PASSED: unmodified jev-reranker ranked from the XERJ node

Everything on this page: the pilot · the full run · the FiQA run · the AG News vote · the gate · the research narrative · measured 2026-09-19..20 · engine at the wire-compat branch plus the WAND commit cf6a9989, ES-YAML conformance 1371 passed / 0 failed on that branch (canonical: 1366 / 1369 today) · related: the benchmarks hub · the rerank stage, documented · the ES scorecard.