Can I route tickets or filter spam from labelled history, without an LLM call?

TL;DR — If you already have labelled examples, you can route and filter text with no LLM or API call. Index the examples, fetch the 10 nearest for each new item, and vote. Measured: 0.933 accuracy on 77-way intent routing and 0.983 on SMS spam. The 0.933 figure needs --embed-mode neural, which runs one local MiniLM embedding per item; the spam figure is plain BM25, with no model of any kind.

The method

A judge model answers typed questions: pick one of N, or yes-or-no with a probability. Where a team already has labelled history, a search index can answer the same shape of question.

1. Index every labelled example, with the label in a keyword field.
2. For a new item, fetch its 10 nearest neighbours.
3. Take a rank-weighted vote over the neighbours' labels.
4. Report the winning label, and its share of the vote as the confidence.

No model runs per decision on the BM25 arm. On the neural arms one local embedding runs per item, on the node, with no API call.

Intent routing: Banking77

Banking77 is a public dataset of banking support messages with 77 intent labels. The run indexed 10,003 labelled examples and classified 1,000 held-out items.

Spam detection: SMS spam

The SMS spam dataset is a public yes-or-no task. The run indexed 4,000 labelled messages and classified 1,000 held-out ones.

Plain BM25 over 4,000 examples reached 0.983 in under a millisecond per message. The neural arm did not improve accuracy here, and it cost far more time.

Which embedder these numbers need

The BM25 rows use no embedder, so they hold on a default XERJ node. The MiniLM and hybrid rows were measured with --embed-mode neural and the all-MiniLM-L6-v2 model on CPU.

XERJ's default embedder is lexical feature hashing. It has no model in it. Do not read the MiniLM or hybrid rows as what a default node scores.

The confidence gate is the point

An expected calibration error of 0.012 means the reported confidence is usable: averaged over confidence bins, the gap between the confidence the vote reported and the accuracy observed at that confidence was about one point in this run.

That makes the confidence a gate. On Banking77, the MiniLM arm decided 86.9% of items at confidence 0.8 or higher, and was right on 0.979 of them. Only the remaining items need a judge model or a person.

This is the economics of a two-stage system. A free local stage takes the bulk, and the paid stage sees only what is uncertain.

When to call a judge model instead

Call a judge model when there is no labelled history. This method is not zero-shot. A new policy or a new category has nothing to vote from, and a judge model needs no examples.

Call one also for the low-confidence share. XERJ's rerank search stage can send documents to an external judge when an operator has set a provider key. XERJ did not run any judge model on these two datasets. It claims nothing about how one would score on them.

What this run does not show

Two public datasets, one run each, 1,000 held-out items per dataset, k fixed at 10. The held-out split uses a seeded shuffle. The latency was measured on a machine that was also compiling, so treat it as an upper bound.

The gap between the two datasets' latencies is query length. Short banking messages embed in a few milliseconds, and longer SMS bodies take tens.

FAQ

Can I classify text without calling an LLM?

Yes, when you have labelled history. Index the labelled examples, fetch the 10 nearest for each new item, and take a rank-weighted vote over their labels. No model call is made per decision.

How accurate is nearest-neighbour classification on a real dataset?

On Banking77, a 77-way intent dataset, XERJ measured 0.819 with BM25 and 0.933 with MiniLM neighbours. On SMS spam it measured 0.983 with BM25 alone. The MiniLM figures need --embed-mode neural.

Can I trust the confidence value?

In this run, yes. Expected calibration error was 0.012 on Banking77 and 0.009 on SMS spam for the MiniLM arm: averaged over confidence bins, the reported confidence and the observed accuracy differed by about one point.

Does this work with no labelled examples?

No. This is not zero-shot. A new category or a new policy has no history to vote from, and that case needs a judge model or a person.

Do I need a neural embedder for this?

Not for spam. BM25 alone reached 0.983 on SMS spam with no embedder at all. For 77-way intent routing the neural arm added about 11 points of accuracy over BM25.

Did XERJ compare this against an LLM judge?

No. XERJ did not run any judge model on these datasets and claims nothing about how one would score.

Evidence

• Banking77, 77-way intent routing, 10,003 labelled examples indexed, 1,000 held-out test items, k = 10: accuracy 0.819 for BM25, 0.933 for MiniLM neighbours and 0.937 for hybrid RRF; expected calibration error 0.089, 0.012 and 0.052; 50.6%, 86.9% and 79.2% of items decided at confidence 0.8 or higher, at accuracy 0.996, 0.979 and 0.989; 2.2, 8.6 and 12.4 ms per item. — benchmarks/decisions-as-retrieval/results/run-2026-09-18.txt
• SMS spam, yes-or-no detection, 4,000 labelled examples indexed, 1,000 held-out test items, k = 10: accuracy 0.983 for BM25, 0.982 for MiniLM neighbours and 0.989 for hybrid RRF; spam F1 0.944, 0.940 and 0.962; expected calibration error 0.017, 0.009 and 0.015; 0.9, 74.5 and 67.2 ms per item. — benchmarks/decisions-as-retrieval/results/run-2026-09-18.txt
• The run was made on 2026-09-18 with xerj v1.0.0-rc.74 and --embed-mode neural using all-MiniLM-L6-v2 on CPU, with a seeded shuffle for the held-out split, on a machine that was also compiling. — benchmarks/decisions-as-retrieval/README.md

Related

• How do I stop an agent memory store from filling with near-duplicate notes?
• How much better is hybrid search than BM25? Measured nDCG numbers
• How do I rerank search results with a calibrated relevance judge?
• When should I turn on neural embeddings, and when is the lexical default enough?