BM25 vs vector search: which one should you use?
BM25 vs vector search: which should you use? BM25 scores the exact words a document shares with the query, so it is best for names, codes and error strings. Vector search compares meaning, so it finds paraphrases. Most search over mixed text works best with both, merged as hybrid search.
TL;DR BM25 finds documents that share words with your query and favors rare words. Vector search finds documents with a similar meaning, even with no shared words. Use BM25 for exact names and error codes, vectors for paraphrases, and both together when queries mix the two.
Two ways to look up "server unreachable"
Act as a coding agent. Read https://xerj.org/llms.txt, install XERJ, start a local node with --embed-mode neural, index a folder of my docs with xerj autoindex, then write 10 test questions (5 with exact names or error strings, 5 written in other words) and report which ones BM25, knn and hybrid each answered in the top 3.
curl -fsSL https://xerj.org/get | sh
Install the single XERJ binary.
curl -s -XPOST 'http://127.0.0.1:9200/docs/_search' -H 'content-type: application/json' -d '{"query":{"match":{"body":"connection refused"}},"size":5}'
A BM25 query. No model needed.
The short version
| BM25 (keyword) | Vector search (semantic) | |
|---|---|---|
| Matches on | Shared words | Closeness of meaning |
| Needs a model | No | Yes, an embedder |
| Exact IDs, error strings | Strong | Can blur near tokens |
| Paraphrases, synonyms | Misses them | Strong |
| Empty results | Yes, when no word is shared | Never; always returns the nearest |
| Shows why a hit matched | Yes, you can see the words | Not directly |
Try it
Type a query, or pick one. The left column is real BM25. The right column is a tiny toy, so you can see the idea without a model.
BM25 next to a toy vector search
Six lines to search:
- Error: connection refused on port 9200
- The agent cannot reach the search server
- Restart the server if the index looks stuck
- Give a coding agent memory across sessions
- Port 9200 is the default search API port
- My assistant forgets what we discussed yesterday
With JavaScript off, here are BM25 scores a local XERJ node returned on these lines:
- "connection refused": line 1 at 3.3007
- "server unreachable": line 2 at 1.0395, line 3 at 0.9829
- "agent memory": line 4 at 2.5948, line 2 at 1.0395
What to look for:
connection refusedis exact text from line 1. BM25 gives it a clear lead.network downandbot remembers last weekshare no word with any line. BM25 returns nothing. The vector side still finds a sensible line.- The vector side always returns 3 lines, even when the third is a poor match. A vector search has no natural "no result".
How BM25 scores a line
For each query word that appears in a line, BM25 adds:
idf × tf × (k1 + 1) / (tf + k1 × (1 - b + b × length / average length))
tfis how often the word appears in the line.idfis high for rare words and low for common ones. "refused" is in one line, so it counts more than "server", which is in two.k1 = 1.2stops a repeated word from adding score forever.b = 0.75gives shorter lines a small boost.
Those are the values in XERJ. The toy above uses the same formula, and on these six lines it gives the same scores as a real XERJ node, to 4 decimals.
How vector search scores a line
A model reads each text and outputs a list of numbers, a vector. Texts with a similar meaning get vectors that point the same way. The query is turned into a vector too, and the search returns the lines whose vectors point closest, measured by cosine similarity.
The toy uses 5 numbers per word that we set by hand. Real models such as all-MiniLM-L6-v2 output 384 numbers per text and learn them from data.
Recorded numbers
Two public BEIR sets, run on one XERJ node with all-MiniLM-L6-v2.
| Dataset | BM25 nDCG@10 | Vector nDCG@10 | Hybrid RRF | Queries with no BM25 hit |
|---|---|---|---|---|
| SciFact, 300 queries | 0.6572 | 0.6764 | 0.6993 | 0 |
| NFCorpus, 323 queries | 0.3016 | 0.3291 | 0.3448 | 25 |
The 25 empty NFCorpus queries are single words such as "okra" or "Zoloft" that appear in no document. Only the vector side could answer them. The nDCG page has the method.
Which should you pick?
- Logs, code, configs, ticket IDs: start with BM25.
- Support questions, notes, chat history: add vectors.
- An agent that asks both kinds of questions: use hybrid search.
Our vector database vs full text search page covers the storage side of the same choice.
How to try it with XERJ
Use the Copy the prompt button and give it to your coding agent. It installs XERJ, indexes a folder and tests both kinds of question. BM25 works with no setup. Start the node with --embed-mode neural to get real vectors from all-MiniLM-L6-v2. The model downloads once, about 90 MB, and runs on your machine.
Sources checked
All checked on 2026-10-30.
- BM25 formula and defaults:
engine/crates/xerj-fts/src/bm25.rsin the XERJ repository. - Benchmark numbers:
benchmarks/beir-hybrid/results/scifact.txt,nfcorpus.txtandnfcorpus-lexical-gap.txt. - Toy scores: a local XERJ v1.0.0-rc.97 node, 6 queries on the six lines above.
FAQ
What is the difference between BM25 and vector search?
BM25 counts shared words and weighs rare words higher. Vector search turns text into numbers with a model and returns the closest texts by meaning, even with no shared words.
Is vector search always better than BM25?
No. On the recorded BEIR SciFact run BM25 scored 0.6572 nDCG@10 and MiniLM vectors 0.6764. That is close. BM25 is also better at exact tokens like IDs and error codes.
Why does BM25 return no results?
BM25 only scores documents that share a word with the query. In the NFCorpus run, 25 of 323 queries shared no word with any document, so BM25 returned nothing for them.
Does BM25 need a model or GPU?
No. BM25 is arithmetic over word counts. Vector search needs an embedding model to turn both the documents and every query into vectors.
What are k1 and b in BM25?
k1 controls how fast repeated words stop adding score. b controls how much long documents are penalized. XERJ uses k1 = 1.2 and b = 0.75.
Can I use both?
Yes. That is hybrid search: run both and merge the ranked lists, often with Reciprocal Rank Fusion.