Playbook · Vector & RAG

Index vectors on one box, query with hybrid BM25 + kNN fusion, and use the dashboards to watch recall, latency, and cache hit rate live.

Schema

$ curl -sX PUT http://localhost:8080/v1/indices/docs \
    -H 'Content-Type: application/json' \
    -d '{
      "fields": {
        "@timestamp": "date",
        "doc_id":     "keyword",
        "chunk_id":   "keyword",
        "text":       "text",
        "title":      "text",
        "tags":       "keyword",
        "embedding":  { "type": "dense_vector", "dims": 1536, "metric": "cosine" }
      }
    }'

Ingest with embeddings

Configure an embedding endpoint in [embedding] and XERJ will call it inline during turbo-ingest:

[embedding]
default_endpoint = "https://api.openai.com/v1/embeddings"
default_model    = "text-embedding-3-small"
batch_size       = 64
timeout_ms       = 5000

Or generate vectors client-side and pass them in the NDJSON body directly.

Pure kNN retrieve

{
  "knn": {
    "field":      "embedding",
    "query_vector": [0.12, 0.08, -0.31, ...],
    "k":          20,
    "num_candidates": 200
  }
}

Hybrid · BM25 + kNN fusion

One request, one planner pass, one round trip. hybrid is a query clause, so it goes inside query, and each entry of queries wraps its clause in query (an optional weight sits beside it for "fusion": "linear").

{
  "query": {
    "hybrid": {
      "fusion": "rrf",
      "queries": [
        { "query": { "match": { "text": "kernel panic after kernel 6.1 upgrade" } } },
        { "query": { "knn": { "field": "embedding", "query_vector": [0.12, 0.08, -0.31], "k": 50 } } }
      ]
    }
  },
  "size": 10
}

Semantic search · embed at query time

Skip the client-side embedding step. Map a field as semantic_text and XERJ embeds both the document at ingest and the query at search time — with the built-in lexical feature-hashing embedder by default, or with the endpoint configured in [embedding] above. The clause is semantic; it takes field and query, and k defaults to 10.

$ curl -sX PUT http://localhost:9200/docs/_mapping \
    -H 'Content-Type: application/json' \
    -d '{ "properties": { "summary": { "type": "semantic_text" } } }'

{
  "query": {
    "semantic": {
      "field": "summary",
      "query": "how do I rotate api keys without downtime",
      "k":     10
    }
  }
}

Dashboards

The playground has two RAG-specific views:

• VECTOR · INDEX — resident set size, vector-index stats, quantization savings, recall at k. (Unfiltered kNN is HNSW-served with exact rescoring — measured recall@10 1.00 on the official bench query; filtered kNN and SQ8 fields run the exact scan, where recall reflects quantization only.)
• RAG · QUALITY — hit rate by intent, cache hit, top retrieved documents.

Source · engine/crates/xerj-engine/src/index.rs (run_knn_hnsw + exact fallback)