A BM25 VOTE TIES JEV ON SPAM
AND BEATS IT ON INTENT

What we compared

Jev is a decision model. You send it a question and a payload, and it returns a typed answer instead of prose, which is a genuinely useful shape for anything that has to branch on the result.

It also raises an uncomfortable question about our own product. If you already hold a pile of labelled history, do you need a model to classify the next item, or can you simply look it up?

So we built the dumbest competitor we could think of: retrieve the ten nearest labelled examples from an index, let them vote, and return the winning label. No training, no fine-tuning, no inference. We then ran it against Jev on two public datasets with five hundred held-out items each, scoring every arm with exact McNemar tests and bootstrap confidence intervals so that a difference of a point or two could not be mistaken for a result.

One thing to be precise about, because our own rules require it: rows labelled semantic or hybrid ran --embed-mode neural with all-MiniLM-L6-v2 in-process on CPU. XERJ's default embedder is lexical feature hashing, and the BM25 rows used no model whatsoever. Everything local still costs zero API tokens either way.

On spam, nobody wins

Both systems get essentially every message right, and the difference between them is noise.

Paired on the same items, hybrid and Jev disagree on ten of them and split those six to four, which an exact McNemar test scores at p = 0.75. There is no detectable difference in accuracy between a hosted frontier decision model and simply counting which labels turn up near your query. The gap that does exist sits in the other two columns, where the local arm is roughly thirty-five times faster and the bill is zero.

On 77 classes, retrieval wins outright

We expected the fine-grained task to be where a hosted model pulled ahead. It went the other way, and not by a little.

Paired, the two disagree on eighty-five items and Jev loses sixty-five of them, which is decisive at p < 0.0001. The best local arm also calibrates better by a factor of three on expected calibration error, so unlike the hosted arm its confidence is worth thresholding on rather than merely sorting by.

Why would a vote beat a model here? Because seventy-seven intents in a banking app are not a general reasoning problem, they are a vocabulary problem, and that vocabulary is already sitting in the index. A model that has never seen this bank's phrasing has to infer the whole taxonomy from a list of option strings, while the index holds ten thousand examples of how real customers actually word each one.

How much history you need before this works

This is the number we would most want if we were reading someone else's post, so here it is.

Twenty-five labelled examples per class. Below that line it is worth paying for the model, and above it you are paying a vendor to do worse than your own archive. Most support teams crossed that threshold years ago without noticing, which is the part of this result that seems genuinely interesting.

Where Jev beat us, and why

We nearly published a much better-looking number here. Our first DBpedia run dumped the class labels as bare integers, so Jev was scoring at chance and the result looked like a rout — until we read our own harness and found the bug. With real class names the comparison flipped completely, and the honest version is the one above.

There is a second loss worth naming, and it is structural rather than incidental. Ask a question whose answer is not in the label vocabulary at all and retrieval scores zero, because a vote can only ever return a label it has already seen; Jev answers the same question zero-shot with the best calibration we measured anywhere. A vote is not a reasoner, and we should stop pretending that the two substitute cleanly for one another.

The comparison found two defects in our own code

Running a competitor properly is the cheapest code review available, and two of ours failed under it. The first is an instruction bug, which showed up when we tested a four-step ladder running from "Is this spam?" up to a multi-clause policy and the two systems moved in opposite directions.

The cause is that our implementation concatenates the instruction into the retrieval query. A longer instruction is better guidance for a model and steadily worse query text for an index, because the instruction is a question rather than something to search for, which is filed as #1000.

The second defect is quieter and worse: a structured state payload was being silently discarded, so the endpoint returned confident answers computed from an empty context and nothing ever errored. That one is #1001.

What we are not claiming

Both datasets are public and may sit in Jev's training data. That biases the comparison in Jev's favour, which makes the Banking77 result conservative rather than inflated, but it is still a real limit on what two datasets can tell you.

The runs used one vendor model version over a single week, sampled once per item, on a shared machine whose load average wandered between eight and twenty-two. Local latencies are therefore lower bounds, and Jev's include a network path nobody outside TypeSafe can see.

Every item was a short English text with a single label, in one language. Nothing here shows that the result transfers to long documents or multilingual corpora, and that claim stays unmade until somebody measures it.

The full write-up, every arm and the per-item predictions live in the study. The endpoints are /v1/systemone and /_decide, both documented in the docs, and the retrieval side of the story continues in our earlier post on reranking and on the benchmarks page. If you would rather check the claim than read about it, the shortest path is the quickstart, an index recipe and your own labelled archive.