Agent memory

Use case: your agent learns things during a session — user preferences, project facts, decisions — and needs to remember them on a later turn to answer well. The usual answer is to bolt a separate vector database next to your search stack. XERJ folds that job into the engine you already run: a namespaced, offline agent-memory API (/_memory/{ns}) that stores text + an embedding + metadata and recalls the most relevant memories by vector similarity or plain text, with metadata filtering, forgetting, and hard per-agent isolation.

This recipe builds a tiny simulated coding-assistant agent (no LLM required) that:

1. stores what it learns as memories (POST /_memory/{ns}),
2. recalls the right memory to answer a new question — semantically (kNN) and lexically (BM25),
3. does a metadata-filtered recall ("only surface team preferences"),
4. forgets a memory that is no longer true, and
5. proves multi-namespace isolation: a second agent can never see the first agent's memories.

Everything below was run end-to-end against a live XERJ. No pip installs — the example is Python 3 stdlib only.

Why this replaces a bolt-on vector DB

A memory namespace is just a reserved XERJ index (.xerj-memory-{ns}) under the hood, so recall reuses the same dense_vector/HNSW kNN, BM25, and metadata-filter paths that serve the rest of the engine. That means:

• One system to run and back up. No second datastore, no sync job keeping a vector index consistent with your source of truth.
• kNN and BM25 in the same store. Recall by embedding similarity, by keyword, or filter both by metadata — no glue code.
• Namespaces are physical isolation. Each agent (or tenant, or user) gets its own backing index; a recall in namespace A literally cannot read namespace B.
• Offline and zero-config. You supply the vectors (from your own model); XERJ never phones out to embed.

The memory API, in four calls

_recall picks the path from the body: a vector runs kNN; otherwise query runs BM25 over the text; an empty body returns recent memories (match_all). filter is a normal ES query clause applied as a bool filter, so it narrows without affecting the score. Recall of an unknown namespace returns {"hits": []} — that is how isolation stays clean.

Storing a memory (raw wire)

curl -XPOST localhost:9200/_memory/agent-ada -H 'content-type: application/json' -d '{
  "text": "The team prefers Rust over Go for new backend services.",
  "vector": [0.12, 0.0, 0.34],
  "metadata": {"kind": "preference", "topic": "language"},
  "id": "demo-1"
}'

{"created":true,"id":"demo-1","namespace":"agent-ada"}

Recalling by vector, filtered to preferences (raw wire)

curl -XPOST localhost:9200/_memory/agent-ada/_recall -H 'content-type: application/json' -d '{
  "vector": [0.12, 0.0, 0.34],
  "k": 2,
  "filter": {"term": {"metadata.kind": "preference"}}
}'

{"hits":[{"id":"demo-1",
          "metadata":{"kind":"preference","topic":"language"},
          "score":1.0,
          "text":"The team prefers Rust over Go for new backend services."}],
 "namespace":"agent-ada"}

Note metadata.kind filters straight out of the box — nested metadata subfields are queryable with a normal term clause, no explicit mapping needed.

The full agent

The script below is the whole example. The one thing to understand about it: it carries a tiny self-contained "embedder" (deterministic hashed character trigrams) so the demo needs no model download and cosine similarity is meaningful.

Honesty check. That toy embedder is lexical, not neural: it rewards shared character n-grams (so service/services, prefer/prefers land near each other), but it does not know that "car" ≈ "automobile". For real semantic recall you pass vectors from your own model (OpenAI/Cohere/a local model) — the store and recall calls are byte-for-byte the same. XERJ's own built-in semantic_text embedder is likewise lexical unless you configure an external /v1/embeddings endpoint; the memory API deliberately takes your vectors so quality is your choice.

The full example script is on GitHub — docs/examples/agentic-memory/agent_memory.py (Python 3 stdlib only, no dependencies).

Reproduce it yourself

Start XERJ (single node, no TLS/auth) on its default port and run the script. Nothing to install — the client is Python 3 stdlib only.

# 1. boot XERJ on the default port 9200
./engine/target/release/xerj --insecure --data-dir /tmp/xerj-mem &

# 2. run the agent (defaults to http://localhost:9200)
python3 docs/examples/agentic-memory/agent_memory.py

# Point it at a different node/port with XERJ_URL if you like:
#   XERJ_URL=http://localhost:9481 python3 docs/examples/agentic-memory/agent_memory.py

What a customer should see. The client exits 0 and prints All assertions passed.. The recall-quality line is the headline number, and it is deterministic — the toy embedder and BM25 are fixed functions of this corpus, so the values below reproduce exactly, run to run:

• Semantic (kNN): recall@1 = 83.3% (5/6), recall@3 = 100% (6/6)
• Lexical (BM25): recall@1 = 100% (6/6), recall@3 = 100% (6/6)

The single semantic recall@1 miss is honest and instructive: on "Which region is production deployed in?" the toy trigram embedder scores "Deploys are gated on CI…" (0.790) a hair above the correct "Production runs on AWS…" (0.780) — shared deploy/Deploys character trigrams. The right memory is still in the top 3, and BM25 recalls it first (6/6), which is exactly why the recipe offers both recall paths.

Real output

== Storing what the agent learned this session ==
  stored ada-1    created=True
  stored ada-2    created=True
  stored ada-3    created=True
  stored ada-4    created=True
  stored ada-5    created=True
  stored ada-6    created=True

== Semantic recall for: "What programming language should I use for a new backend service?"
  [0.779] (preference) The team prefers Rust over Go for new backend services.
  [0.741] (fact      ) The payments service owns the Postgres 'ledger' database.
  [0.731] (preference) The team prefers tabs over spaces and a 100-column line limit.

== Lexical (BM25) recall for: "Which region is production deployed in?"
  [3.746] (fact      ) Production runs on AWS in us-east-1; staging is us-west-2.
  [1.010] (fact      ) Deploys are gated on the full integration suite passing in CI.

== Preference-only recall for: "How should I set up the new service?"
  [0.765] (preference) The team prefers Rust over Go for new backend services.
  [0.692] (preference) The team prefers tabs over spaces and a 100-column line limit.
  [0.666] (preference) The team prefers OpenTelemetry for tracing, not vendor SDKs.

== Recall quality over 6 labeled probe questions ==
  semantic (kNN):  recall@1 =  83.3%  (5/6)   recall@3 = 100.0%  (6/6)
  lexical  (BM25): recall@1 = 100.0%  (6/6)   recall@3 = 100.0%  (6/6)

== Namespace isolation ==
  agent-billing recall for a coding question -> ['Invoices over $10k require CFO approval.']
  OK: agent-billing only ever sees its own namespace.

== Forgetting a memory that is no longer true ==

  before forget:
  [0.755] (preference) The team prefers OpenTelemetry for tracing, not vendor SDKs.
  [0.692] (fact      ) Production runs on AWS in us-east-1; staging is us-west-2.
  [0.683] (preference) The team prefers Rust over Go for new backend services.
  forget -> {'forgotten': True, 'id': 'ada-6', 'namespace': 'agent-ada'}

  after forget:
  [0.692] (fact      ) Production runs on AWS in us-east-1; staging is us-west-2.
  [0.683] (preference) The team prefers Rust over Go for new backend services.
  [0.661] (preference) The team prefers tabs over spaces and a 100-column line limit.
  OK: the retracted memory is gone.

All assertions passed.

Read the run top to bottom:

• Semantic recall put the Rust preference on top for a question that never says "Rust" or "prefer" — trigram overlap on backend / service / new pulled it up over unrelated facts.
• BM25 recall answered the region question exactly, no vectors involved.
• Preference-only recall returned three memories, all kind:"preference" — the metadata.kind filter did its job.
• Recall quality over the 6 labeled probes is measured, not asserted by hand: BM25 recall@1 = 6/6, semantic recall@1 = 5/6, both recall@3 = 6/6 — a real number this run computed and printed.
• Isolation held: agent-billing, asked a coding question, only saw its own single finance memory.
• Forgetting removed ada-6 (the tracing preference); the follow-up recall no longer surfaces it.

Notes, limits, and going to production

• Bring your own embeddings. The memory API stores whatever vector you pass and kNN-searches it with cosine similarity. Swap the toy embed() for a call to your model's embeddings endpoint and everything else is unchanged. Keep the vector dimension consistent within a namespace — the backing index's dense_vector field is sized from the first stored vector.
• Vector or text. A memory needs at least text or a vector. Text is always BM25-indexed; a vector additionally enables kNN recall. Recall with vector uses kNN; without it, query uses BM25.
• Recency (opt-in). Recall is pure relevance by default. Pass recency_weight in [0, 1] on _recall to blend recency: the engine over-fetches candidates and re-ranks by (1 - w) * norm_relevance + w * norm_recency (both min-max normalized across the candidate set), so w = 1 surfaces the newest memory and w = 0 is exactly the default relevance order. Verified: two same-vector memories tie on relevance; with recency_weight: 1.0 the newer one is returned first. Useful when an agent's latest observation should win ties.
• Isolation is structural, not a filter. Namespaces are separate physical indices, so there is no query you can write in one namespace that reads another. Use one namespace per agent/tenant/user.
• GET /_memory/{ns} lists up to 100 most-recent entries for debugging/audit; it is bounded and recent-first, not a full export.
• Wire-compatible. Because a namespace is a real index, you can inspect or operate on it with the standard ES-compatible surface if you ever need to — the memory API is a thin, honest adapter over the same machinery.