What is semantic search?
What is semantic search? It ranks passages by meaning, not only by shared words. Each passage and the query become vectors, and rank is by closeness in that space, so a paraphrase can still match. In XERJ, a full-text query uses BM25 by default. Neural or hybrid mode adds a vector arm.
TL;DR Semantic search ranks passages by meaning, usually by turning the query and each passage into vectors and ranking by closeness. A paraphrase can match even when it shares few words with the query.
Lexical match vs meaning match
Same words
A lexical query finds passages that share terms with the question. It misses a passage that says the same thing in different words.
Close vectors
Each passage and the query become vectors. Rank is by closeness in that space, so a paraphrase can still match. This is the idea behind dense passage retrieval.
Default vs neural
XERJ answers a full-text query with BM25 by default. Neural or hybrid mode adds a vector arm. The default embedder is lexical and runs offline, with no API key.
Act as a coding agent. Read https://xerj.org/llms.txt, install XERJ, start a local node, index one folder I point you to, and run one search so I can see a cited passage.
How it works
A lexical query finds passages that share terms with the question. It misses a passage that says the same thing in different words.
A semantic query represents the question and each passage as vectors. Rank is by closeness in that space. Two texts that share an idea can match even if the words differ. Two identifiers that look alike can also sit close together, which is why exact tokens still need a lexical arm.
Dense passage retrieval is the usual name for a trained dual encoder that does this for question answering.
How XERJ uses it
XERJ answers a full-text query with BM25 by default. Neural or hybrid mode adds a vector arm. The default embedder is lexical and runs offline, with no API key. Vectors live in the same index as the lexical data, so you do not need a separate vector database.
The how-to for running both arms is how to run hybrid search in XERJ. Related answers: BM25 vs vector search and vector database vs full-text search.
History and research
A widely cited method for this style of retrieval is Karpukhin and colleagues, Dense Passage Retrieval for Open-Domain Question Answering (EMNLP 2020 / arXiv:2004.04906).
XERJ behaviour follows the product docs, not that paper's numbers. This page does not add a new embedding benchmark.
Related terms
BM25, hybrid search, and reranking.
FAQ
What is semantic search?
It is retrieval that ranks by meaning. The usual method is to embed the query and each passage as vectors, then rank by closeness.
How is this different from BM25?
BM25 needs shared terms. Semantic search can match a paraphrase that uses different words. BM25 stays stronger on exact names and identifiers.
How does XERJ do semantic search?
Neural or hybrid mode adds a vector arm. The default embedder is lexical and runs offline, with no API key. You do not need a separate vector database.
Is this the same as hybrid search?
No. Semantic search is one arm. Hybrid search is what you get when you also run a lexical arm and fuse the two lists.
Related
- Hybrid vs vector vs semantic search
- BM25 vs vector search
- What is hybrid search?
- BM25 vs vector search: which one should you use?
- What's the actual difference between a vector database and full-text search when I have a folder?
- How do I build RAG without a vector database?
- What is syntax-aware search? · same intent: Science
- What is agentic memory? · same intent: Science
- What is context engineering? · same intent: Science