XERJ vs mcp-memory-service: two MCP memory servers, tested
XERJ or mcp-memory-service? With MiniLM, both found 84% of reworded questions. mcp-memory-service found 89.3% of all answers in the top 5 and XERJ semantic recall 94.7%. At default settings mcp-memory-service rejected 4 of 15 updated facts as duplicates. XERJ stores every write unless you ask for dedup.
Key number: mcp-memory-service turns semantic dedup on by default: a new memory 0.85 similar or more to one from the last 24 hours is rejected. (as of 2026-10-31 · source: github.com/doobidoo/mcp-memory-service/blob/main/src/mcp_memory_service/storage/mixins/base.py)
TL;DR mcp-memory-service and XERJ both serve agent memory over MCP and HTTP. Both can use a local MiniLM model; XERJ needs neural mode for it. Recall was close: 89.3% top 5 for mcp-memory-service, 94.7% for XERJ semantic. The big difference was changed facts: mcp-memory-service's default dedup threw away 4 of 15 updates.
What happens to an updated fact
mcp-memory-service, default settings
- Day 1"The Lark API listens on port 8443"Stored.
- LaterSame fact, new portCosine similarity with the old note is above 0.85.
- ResultRejected as a duplicateOnly the old value can come back.
XERJ, default settings
- Day 1Old value storedAs sent.
- LaterNew value storedDedup is off unless the write asks for it.
- RecallBoth can come backA recency weight puts the new one first.
Act as a coding agent. Read https://xerj.org/llms.txt, install XERJ, start a local node with --embed-mode neural, store a fact about this project with POST /_memory/project, store a changed version of the same fact, then recall it with hybrid recall and recency_weight 0.3 and tell me whether the new value came first.
curl -fsSL https://xerj.org/get | sh
Install the single XERJ binary.
xerj --embed-mode neural
Start a local node with the built-in MiniLM embedder.
curl -s -XPOST 'http://127.0.0.1:9200/_memory/project/_recall' -H 'content-type: application/json' -d '{"query":"API port in production","hybrid":true,"recency_weight":0.3,"k":5}'
Hybrid recall that ranks newer memories higher.
Tested side by side
Same 85 memories, same 75 questions, same MiniLM model. Top 5 is the share of questions where the right memory came back in the first five.
| mcp-memory-service 12.4.0 | XERJ v1.0.0-rc.100 | |
|---|---|---|
| Top 5, all questions | 89.3% | 94.7% semantic, 93.3% hybrid |
| Top 1, all questions | 64.0% | 65.3% semantic, 60.0% hybrid |
| Top 5, reworded questions | 84% | 84% |
| New value first, changed facts | 6.7% | 13.3%; 93.3% with recency weight |
| Updates rejected at write | 4 of 15 | 0 |
| Median write | 16.7 ms | 7.2 ms |
| Median query | 16.7 ms | 6.3 ms |
| LLM calls per write | 0 | 0 |
| What you install | 72 Python packages | One binary |
| Storage | SQLite with sqlite-vec, optional cloud sync | Index per namespace |
| License | Apache 2.0 | Apache 2.0 |
Question by question
This starts on the changed facts. A check means the right memory was in the top 5.
XERJ and mcp-memory-service, question by question
| Question | Answer | XERJ hybrid | XERJ recency | mcp-memory-service |
|---|---|---|---|---|
| Does Dana prefer pnpm or npm? | pnpm | 1 | miss | 1 |
| What is the main database for Lark? | PostgreSQL | 1 | miss | 1 |
| What is Dana allergic to? | peanuts | 1 | miss | 1 |
| Which region does the staging cluster run in? | eu-west-3 | 1 | miss | 1 |
| What is Lark's frontend written in? | Svelte | 1 | miss | 1 |
| Where does error tracking for Lark go? | Sentry | 1 | miss | 1 |
| What is Dana's cat named? | Biscuit | 1 | miss | 1 |
| How fast must the test suite finish? | 12 minutes | 1 | 4 | 1 |
| What does the billing service use for card payments? | Stripe | 1 | 5 | 1 |
| What is the Lark mobile app built with? | Flutter | 1 | 5 | 1 |
| What is the Lark design system called? | Pebble | 1 | 4 | 1 |
| Where are Lark's public docs hosted? | Mintlify | 1 | 2 | 1 |
| What is the analytics warehouse? | ClickHouse | 1 | 2 | 1 |
| What is the support inbox address? | [email protected] | 1 | 1 | 1 |
| Which day does Dana's daughter Mia have swim practice? | Wednesdays | 1 | 4 | 1 |
| What is the load balancer in front of Lark? | HAProxy | 1 | 3 | 1 |
| What do background jobs run on? | Sidekiq | 1 | 2 | 1 |
| What is Dana's GitHub handle? | dana-codes | 1 | 2 | 1 |
| What is the Redis cache TTL? | 300 seconds | 1 | 1 | 1 |
| What is the Slack channel for incidents? | lark-fire | 1 | 1 | 1 |
| Where are secrets stored? | Vault | 1 | 2 | 1 |
| What is transactional email sent through? | Postmark | 1 | 1 | 1 |
| What is the monorepo built with? | Turborepo | 1 | 1 | 1 |
| When do database backups run? | 02:00 | 1 | 1 | 1 |
| What is Lark's mascot? | Hugo | 1 | 1 | 1 |
| Which package manager should I use for JS work? | pnpm | 2 | 5 | 1 |
| Does she have any food intolerance I should know about before ordering lunch? | peanuts | 1 | 4 | 1 |
| Who is the boss Dana answers to? | Tomas | miss | 1 | miss |
| Which UI framework powers the web client? | Svelte | miss | miss | miss |
| Where do crash reports and exceptions end up? | Sentry | 1 | miss | 1 |
| What is the name of her pet? | Biscuit | 1 | miss | 2 |
| How does Dana take her coffee? | oat milk | 3 | miss | 2 |
| Which company handles credit card charges? | Stripe | 1 | 1 | 1 |
| What computer does she use at the office? | ThinkPad | 5 | miss | 2 |
| Which toolkit was used for the iOS and Android client? | Flutter | 2 | miss | 1 |
| What style should git log entries follow? | imperative | 5 | miss | 4 |
| Which language does Dana like coding in most? | Rust | 1 | 3 | 1 |
| Is there a legal rule about where user records may be kept? | inside the EU | 1 | miss | 1 |
| What sport does Dana's kid do midweek? | swim | 2 | miss | 3 |
| How long should a call with Dana be? | 25 minutes | 5 | miss | 1 |
| Does Dana eat meat? | vegetarian | 1 | 2 | 1 |
| Which human languages does the product UI support? | Portuguese | 5 | miss | 2 |
| Where do we keep passwords and API tokens? | Vault | 2 | miss | 1 |
| What exercise does she do at the weekend? | 10k | 4 | miss | 2 |
| Which tool maps the JavaScript backends to SQL tables? | Prisma | miss | miss | miss |
| How quickly should a pull request get feedback? | same day | miss | miss | miss |
| What bird represents the brand? | heron | 3 | miss | 1 |
| Which city is home to Dana's sibling? | Porto | 3 | 3 | 1 |
| How slow may search get for the slowest one in twenty requests? | 200 ms | 1 | 3 | 1 |
| Under which open source terms is the code published? | MPL-2.0 | 1 | 1 | 2 |
| What port does the Lark API listen on in production? | 9443 | 2, old first | 1 | miss |
| Which day does Dana's team deploy to production? | Tuesdays | 1 | 1 | 1 |
| Who is Dana's manager? | Tomas | miss | miss | 3, old first |
| What code editor does Dana use? | Zed | 2, old first | 1 | 3, old first |
| Where does the Lark CI pipeline run? | GitHub Actions | 2, old first | 1 | 2, old first |
| What is Lark's search feature backed by? | Typesense | 2, old first | 1 | 2, old first |
| Where are feature flags managed? | Unleash | 2, old first | 1 | 2, old first |
| How long are application logs retained? | 90 days | 3, old first | 1 | 2, old first |
| What timezone does Dana work on? | Berlin | 2, old first | 1 | 2, old first |
| When do auth JWTs expire? | 60 minutes | 1 | 1 | miss |
| What do Lark's Python services use for dependency management? | uv | 2, old first | 1 | 2, old first |
| When is the Lark 2.0 launch? | April 2 | 2, old first | 1 | 2, old first |
| Which floor is Dana's desk on? | 3rd floor | 2, old first | 1 | miss |
| What is the Lark API rate limit per key? | 1200 | 2, old first | 1 | miss |
| Which Kubernetes version runs in production? | 1.33 | 2, old first | 1 | 2, old first |
| What is the staging database for Lark? | MySQL | 1 | 1 | 1 |
| Which package manager does Dana's brother prefer? | yarn | 2 | 1 | 5 |
| What port does the internal admin API listen on? | 7001 | 1 | 1 | 1 |
| Which day does the marketing website deploy? | Fridays | 1 | 1 | 1 |
| What is Priya's cat named? | Pixel | 1 | 1 | 1 |
| How long are search query logs retained? | 7 days | 1 | 1 | 1 |
| What is the mobile app's rate limit? | 100 requests | 1 | 2, old first | 1 |
| What does the data team use for local analysis? | DuckDB | 1 | 1 | 1 |
| What was Dana's old personal laptop? | MacBook | 1 | 1 | 1 |
| Which street is Mia's school on? | Rua das Flores | 1 | 1 | 1 |
When to pick mcp-memory-service
- You want sync across machines through Cloudflare or another copy of the service.
- You want a dashboard, OAuth and scheduled consolidation included.
- You store lots of near duplicate notes and want them filtered for you.
- You are happy to run a Python service.
When to pick XERJ
- Facts in your project change, and you want the new value kept and ranked first.
- You want one binary with no Python environment.
- You want memory in the same engine as your logs, docs and code search.
- You want BM25, semantic and hybrid recall from one API.
Limits on the XERJ side
- mcp-memory-service beat XERJ hybrid on the first result, 64.0% vs 60.0%.
- XERJ is single-node, with no built-in cloud sync or replication.
- The recency weight that ranks new values first lowers overall recall, so use it for facts that change.
See the full AI agent memory benchmark for all seven projects.
Try it with XERJ
Use the Copy the prompt button and paste it into your coding agent. It stores a fact, changes it and checks whether XERJ returns the new value first.
Sources checked
All checked on 2026-10-31.
- mcp-memory-service: README and src/mcp_memory_service/storage/mixins/base.py in doobidoo/mcp-memory-service, commit 0071ab2a, PyPI release 12.4.0. License Apache 2.0 from the GitHub API.
- Test runs and per question results: benchmarks/agent-memory.
FAQ
What is mcp-memory-service?
An Apache 2.0 memory backend for AI agents with a REST API, MCP, a CLI and a dashboard. It stores memories in SQLite with local ONNX embeddings by default, and can sync to Cloudflare or another instance.
Which one recalled more in your test?
On top 5, XERJ semantic found 94.7% and mcp-memory-service 89.3%. On the first result, XERJ semantic was 65.3% and mcp-memory-service 64.0%. Both found 84% of reworded questions.
Why did mcp-memory-service miss changed facts?
Its semantic dedup rejected 4 of our 15 updates, such as a new API port, because they were too close to the old note. Only the old value could come back. You can turn dedup off or pass a conversation_id to skip it.
Which one is faster?
XERJ semantic queries took 6.3 ms median and writes 7.2 ms. mcp-memory-service took 16.7 ms for both. It ran in process and XERJ over HTTP.
Do I need Python for mcp-memory-service?
Yes. It is a Python package. With the SQLite extra it resolved to 72 packages on Python 3.12. XERJ is one binary.
Evidence
- Top 5: mcp-memory-service 89.3%, XERJ semantic 94.7%, XERJ hybrid 93.3%. Top 1: mcp-memory-service 64.0%, XERJ semantic 65.3%, XERJ hybrid 60.0%. Reworded top 5: both 84%. New value first: mcp-memory-service 6.7%, XERJ 13.3%, XERJ with recency weight 93.3%. —
benchmarks/agent-memory/out/summary.csv
Related
- AI agent memory compared: real tests on 7 open-source projects
- XERJ vs agentmemory: which memory for your coding agent?
- claude-mem vs MemPalace vs XERJ: which memory should Claude Code use?
- XERJ vs memvid: one memory file or one local server?
- What is an MCP memory server?
- What is hybrid search?
- XERJ vs ReMe (Alibaba Tongyi Lab): a tested comparison · same intent: agent memory
- XERJ vs Mem0: which memory fits your AI agent? · same intent: agent memory
- What's a simple way to give an agent long-term memory on my laptop without Qdrant? · same intent: agent memory