DOCS·RECIPES·VECTOR QUANTIZATION
RECIPE · 05

Vector quantization

Read this first. scalar8 on XERJ today changes precision, not memory. Scores are computed from 1-byte-per-dimension codes, so the field has the recall profile of int8 — but the serving path still reads the full-precision vector out of _source and quantizes it per query, so nothing gets smaller in RAM. The ingest-time code array that would make it a memory win is tracked in #392. If you came here for a smaller working set, that issue is the one to watch, not this recipe.

The problem

Dense vectors are heavy. A 768-dim float32 embedding is ~3 KB; a million of them is ~3 GB of vector data that has to be resident to serve low-latency kNN. Scale to tens of millions and the vector working set — not the text, not the postings — becomes the thing that decides how much RAM you rent.

The standard fix is scalar quantization: store each dimension in one byte instead of four. The catch everyone worries about is recall — does compressing the vectors quietly wreck ranking quality? That is the question this recipe answers, and the answer is no.

Why XERJ

XERJ lets you opt a dense_vector field into scalar8 (int8) quantization per field. When you do, the kNN serving path scores against 1-byte-per-dimension codes instead of 4-byte floats, while _source still returns the original vectors for retrieval. It's off by default (full float32), so you choose the precision model per field, spelled exactly like Elasticsearch's int8_hnsw (XERJ accepts that name as a compatibility alias — SQ8 fields are served by the exact code scan, not the HNSW ANN path, so kNN on a quantized field stays exact over the codes).

On a real 128-dim corpus the recall cost is negligible: recall@10 = 0.998 against the exact float32 index. That number is computed by the run below, not stipulated.

What that costs in bytes as an encoding is 128 rather than 512 per vector, and the run measures that too — but see the note at the top: XERJ does not hold those codes resident today, so treat it as the size of the encoding, not as a saving you get.

The solution

Opt a field in at mapping time with index_options.type: int8_hnsw:

curl -sX PUT "$XERJ_URL/docs" -H 'content-type: application/json' -d '{
  "mappings": {
    "properties": {
      "title": { "type": "text" },
      "v": {
        "type": "dense_vector",
        "dims": 128,
        "similarity": "cosine",
        "index_options": { "type": "int8_hnsw" }
      }
    }
  }
}'

Index and query exactly as you would a full-precision field — nothing else changes:

curl -sX POST "$XERJ_URL/docs/_search" -H 'content-type: application/json' -d '{
  "knn": { "field": "v", "query_vector": [0.12, 0.08, -0.31, "..."], "k": 10 }
}'

The scores come back slightly different from an exact float32 index (that's the quantization at work — a query that exactly matches a stored vector scores ~0.99999 instead of 1.0), but the ranking is the same.

Try it

docs/examples/vector-quantization/quant_demo.py (the mirrored recipes/vector_quantization.py runs the same demo) embeds the 40 real KB articles into 128-dim vectors, indexes the same vectors into a float32 index and a scalar8 index, and prints the side-by-side top hits, the measured recall@10, and the measured byte footprint of each encoding:

$ python3 docs/examples/vector-quantization/quant_demo.py
embedded 40 real KB articles into 128-dim vectors

indexed into `vq-none` (float32) and `vq-scalar8` (int8_hnsw / scalar8)

query: 'how do I stop an agent's context window from overflowing?'

── float32 (exact)
    0.67958  Long-context windows do not replace memory
    0.60029  p95 latency budgets for interactive RAG agents
    0.59712  SOC 2 controls that apply to vector workloads

── scalar8 (quantized)
    0.67938  Long-context windows do not replace memory
    0.60021  p95 latency budgets for interactive RAG agents
    0.59731  SOC 2 controls that apply to vector workloads

recall@10 (scalar8 vs float32 ground truth): 0.998
encoding size over 40 vecs: float32 = 20480 B (512 B/vec)  →  scalar8 = 5120 B (128 B/vec)  (4.00x smaller)

OK — recall preserved through 1-byte-per-dim codes. `_source` still holds
the originals. scalar8 changes precision, not resident memory (issue #392).

The encoding-size line is a real measurement: the run encodes every corpus vector as float32 bytes (struct) and as int8 codes and compares the actual byte totals — 20480 B vs 5120 B, exactly 4.00×. It is the cost of the two encodings, measured in the client. It is not a measurement of XERJ's resident footprint, and XERJ does not currently realise it as one (#392).

Reproduce it yourself

# 1. Start XERJ (dev mode, default ES-compat port 9200)
xerj --insecure --data-dir ./data &

# 2. Run the demo (stdlib-only Python 3, no packages, no API keys)
python3 docs/examples/vector-quantization/quant_demo.py

XERJ_URL overrides the server (default http://localhost:9200); XERJ_KB overrides the KB path (default: auto-discovered demo/data/ai_kb.ndjson). The embedder and corpus are deterministic, so a customer should see exactly:

These numbers are stable run-to-run (verified across repeated runs — no variance); the printed kNN scores are likewise identical each run.

Notes and limits