Install XERJ (docs: https://xerj.org/llms.txt), index this project's sources, and set up reference coding: clone and index the open-source repos closest to what we're building, and search how they solved a problem before writing code.
One paste — your agent installs XERJ, indexes your project and the open-source repos worth learning from, and looks implementations up instead of re-deriving them.
Two things the agent has to get right, both stated in
llms.txt: indexing needs a running node
— xerj --insecure --data-dir ./.xerj-data & comes
first, or xerj autoindex exits 1 — and
cloning reference repos downloads real data onto your machine,
so have it name the repos and estimate the job before it starts. XERJ's own
doctrine is estimate first, ask before long jobs.
A coding agent hitting an API it hasn't memorised invents method names,
fails to compile, and loops — spending output tokens, the
expensive kind, on every lap. Grep doesn't save it: grep tells the agent
where to look, and the recovery is still reading source into context —
up to 1.06M input tokens on one corpus in our
measurements. Reference coding flips the loop: clone the open-source
repos closest to what you're building,
xerj autoindex them once, and the agent
retrieves the exact definition — with its contract — before writing.
Measured (8 tasks across 4 languages, 16 runs per arm, real
claude -p token counts): 2.7× fewer
output tokens than grep-driven Claude Code at the same 16/16
solve rate, 26× fewer than working from memory
(260,916 → 9,982), 2.1× cheaper ($1.58 vs $3.27). In a
companion run on a Rust library the model had never seen:
9/9 with retrieval vs 0/9 from memory. The loop is four
commands and no extra tooling — it is exactly what
llms.txt walks your agent through when you paste the
prompt above.
$ xerj --insecure --data-dir ./.xerj-data & # a node, running $ git clone --depth 1 https://github.com/spacejam/sled ref/sled $ xerj autoindex ref/sled # index once — tree-sitter AST fields $ xerj search "fsync the WAL segment on rotation" → the exact function, its file:line and contract — one plain-English line in, a passage to read out
One binary, no wrapper scripts: xerj search is a
client — it detects the running node and queries it, ranking symbol
definitions first (defs^3) so you get the
function, not a grep list. For a whole team, the repeatable-corpus helpers
(xc-corpus.sh / xc.py)
still ship under
tools/xerj-code/:
they add a corpus definition — a few hundred bytes of pinned commit SHAs and
licences, no source — so xc-corpus.sh --from
rebuilds the exact same commits on another machine.
THE FULL CASE STUDY — EVERY NUMBER, EVERY TASK, THE EXACT PROMPT →·WHERE THE USERS-REPORT ~5× FIGURE COMES FROM →
The fastest way to make data useful to an AI agent is to not write a
pipeline at all. xerj autoindex <folder>
— a subcommand of the same binary — walks the tree, sniffs every file's
format by content (extensions are never trusted) across
13 format families — JSONL, JSON, dialect-sniffed CSV, logs, SQL dumps,
SQLite, PDF, DOCX, HTML, XML, YAML, plain text, gzip — infers field
types and date encodings from the data itself, writes explicit mappings,
and streams everything in with idempotent IDs. Junk files are recorded,
never fatal. It ends by writing a catalog index — the data map — so the
agent's first question, "what is even in here?", is answered by
the engine. Real captured run:
$ xerj autoindex ./sample autoindex: 4 files (0 MB) under /tmp/xerj-pubpass/sample phase A: sniffing + sampling 4 files… phase A: 3 datasets inferred, 1 junk/skipped files phase B: indexing 3 files with 8 workers → http://localhost:9280 done in 0.2s — 3 datasets, 5801 records live, 0 junk records, 1 junk/skipped files ax-logs 5000 docs ax-exports 800 docs ax-docs 1 docs next: `xerj autoindex map --url http://localhost:9280` for the data map; search via GET /ax-*/_search
CAPTURE NOTE · THAT RUN'S NODE WAS STARTED ON A NON-DEFAULT PORT, SO THE OUTPUT ECHOES :9280. THE DEFAULT ES-COMPAT PORT IS :9200 — USE http://localhost:9200 UNLESS YOU CHANGED IT. THE TRANSCRIPT ABOVE IS REPRODUCED UNEDITED.
Honesty first: in a controlled 10-question exam on that 518 MB corpus, a XERJ-backed agent scored 9 correct + 1 partial vs a fair grep/python baseline's 10/10 — a tie on accuracy, not a rout. What XERJ wins is structural: a full corpus inventory in 4 API calls, sub-second aggregations over millions of rows, and uniform access to SQLite, DOCX, gzip, and decimal-comma CSV through one API — the advantages that matter at scale, over remote/API-only access, and under repeated querying. The pipeline is streaming and resumable, verified on multi-GB corpora.
Every operation an agent needs is a plain HTTP call on :9200 — no SDK, no signup, no external embedding key. Store what an agent learns, recall it later by meaning (the built-in embedder is lexical hash-based — honest hybrid lexical+vector retrieval, not neural understanding). Each namespace is physically isolated, so agents never read each other's memories. Real run against an empty XERJ:
$ curl -sXPOST localhost:9200/_memory/agent-demo \
-H 'content-type: application/json' \
-d '{"text":"The user prefers metric units and a dark UI theme.","metadata":{"kind":"preference"}}'
{"created":true,"id":"77eff57b-e432-431c-8b49-8a16b33ab551","namespace":"agent-demo"}
$ curl -sXPOST localhost:9200/_memory/agent-demo/_recall \
-H 'content-type: application/json' \
-d '{"query":"what display settings does the user like?","semantic":true,"k":1}'
{"hits":[{"id":"77eff57b-e432-431c-8b49-8a16b33ab551",
"score":0.655571460723877,
"text":"The user prefers metric units and a dark UI theme."}], // metadata trimmed
"namespace":"agent-demo"}
xerj autoindex <folder> — sniffed formats, inferred mappings, a self-describing data map, resumable idempotent ingest.
The /_memory REST API — store, recall by meaning / keyword / vector, filter, forget, per-agent isolation.
First-class documentation, each validated end-to-end against a live XERJ — RAG, hybrid search, log analytics, anomaly detection, ES migration.
Every feature exists to cut the thing agents actually spend: context. Reading a file costs a context window; asking an index costs kilobytes. XERJ puts everything an agent needs to ask — code, documents, vectors, memory — behind one query surface.
/_memory REST API — store, recall by meaning / keyword / vector, filter, forget, physically isolated per agent — and a /_graph knowledge layer with evidence on every link.gitignore, records junk instead of crashing on it, and resumes incrementallyCEO for Xerj / [Your Name]