Answers

Direct answers about XERJ search, indexing and agent workflows, with evidence links and practical guidance for AI builders.

Agent access and memory

• Can ChatGPT search a folder on my laptop, or do I need something else? — xerj mcp is a stdio MCP server that proxies 10 tools to a running XERJ node, so an agent reads an indexed local folder instead of opening files itself.
• I need my agent to find a passage and cite the file, not dump the whole tree. How? — Index the repository with xerj autoindex, then call xerj_search through xerj mcp. One captured call returned the file and line for 2 symbols without opening a file.
• Should an agent use API search or vector search? — One question through 4 XERJ MCP tools returned 3 different result sets. Hybrid RRF returned 6 documents and merged both lists on the default lexical embedder.

Agent memory: link retrieval

• Should agent memory use a knowledge graph? — XERJ builds a bounded link index over indexed documents, not a graph database. Our capture found 7 edges, answered 1 and 2 hops, and refused 3.

Agent memory: model dependency

• How do I stop an agent memory store from filling with near-duplicate notes? — Send dedup true on the store call and XERJ returns the existing id instead of a second document. The write path runs no model and made no outbound call.

Agent memory: provenance

• How can agent-memory links retain evidence? — Every detected link in XERJ carries the exact line that taught it. All 7 edges in our capture had an evidence quote, and unlink wrote a tombstone.

Agent memory: security

• How do I give two agents private memory on one laptop? — XERJ keeps agent memory on local disk and made 0 outbound connections in our capture. Roles are stored but not enforced, so scoped API keys do the work.

Agent memory: setup

• How do people store user preferences so an agent can recall them next week? — XERJ keeps agent memory in a reserved index inside the data directory. We stored two memories, restarted the node, recalled both, then forgot one by id.

Agent memory: storage

• How do I set the MCP memory storage path? — XERJ has no separate memory path. Agent memory lives in the node data directory as .xerj-memory-{namespace}, and the MCP server stores nothing itself.

Agent memory: topology

• What's a simple way to give an agent long-term memory on my laptop without Qdrant? — Our capture ran agent memory on one XERJ process with three loopback sockets, no second data service, and a peak resident size of 100,928 kB.

Capabilities and limits

• Does XERJ support S3 storage, alerting or custom plugins yet? — Since v1.0.0-rc.77 the index CAN live in an S3-compatible bucket; since rc.78 record packs are real; since rc.80 _watcher evaluates exactly one watch shape and refuses every other with 501. Plugins stay built-in.

Code security

• How do I audit a PHP codebase with an index instead of grepping 1,400 files? — The published WordPress core audit indexed the tree as queryable facts, then read only the survivors. Core came back hardened, and that negative is the result.

Coding agents: reference retrieval

• How do I set up reference coding so the agent retrieves a mechanism before writing code? — Clone the library you are about to write against, index it with xerj autoindex, and retrieve the passage first. The published study, with the tracks where it loses.
• xerj corpus index --fresh failed. How do I rebuild a corpus index? — xerj corpus index --fresh builds a replacement index beside the old one, verifies it has records, switches the state file, and only then retires the old index. A failed build costs nothing.

Decisions as retrieval

• Can I route tickets or filter spam from labelled history, without an LLM call? — Index labelled examples and classify new text by its nearest neighbours. Measured: 0.933 accuracy on 77-way intent routing, 0.983 on SMS spam, no LLM or API call.

Embeddings: cost

• How do I reduce embedding API cost? — XERJ embeds documents inside the node by default, so a captured lexical run made 0 outbound calls. Proxy mode sent exactly 2 calls for the same 100 documents.

Embeddings: local execution

• I want local search that works offline, no cloud embeddings. What are my options? — XERJ embeds locally in two ways. The default lexical embedder needs no model at all, and --embed-mode neural loads a cached MiniLM model on the CPU.

Embeddings: selection

• When should I turn on neural embeddings, and when is the lexical default enough? — Embeddings only helped when we opted into the neural embedder. On XERJ's lexical default, kNN promoted coffee storage above the synonyms for the query car.

Files and formats

• How do I find every place a config key is set across YAML and XML in a repo? — XERJ autoindex indexes YAML and XML config files as full-text documents, maps 9 fields including body and title, and answers one bool query across both.
• How do I full-text search a SQLite database I just copied onto disk? — xerj autoindex reads a SQLite file read-only and turns each table into an index. A captured WAL-mode run indexed 400 of 450 committed rows and changed no bytes.
• How do I index a CSV with many columns? — xerj autoindex infers a mapping for a wide CSV with no schema work. A captured 120-column file produced 128 mapped fields and an exact terms aggregation.
• How do I index Markdown into the Elasticsearch API? — XERJ autoindex indexes Markdown as the txt-prose family and answers the Elasticsearch API, but it maps no headings array and keeps the hash in the title.
• How do I run syntax-aware code search? — XERJ indexes source files with a defs field and a symbols array, so one query finds declarations with kind, name and line while another finds raw text.
• How do I search a .csv.gz or .json.gz without unzipping it first? — Gzip is a wrapper, not a family. autoindex unwraps it and sniffs the inner CSV or JSON, so a compressed export becomes typed fields rather than one opaque blob.
• How do I search a Confluence export? — Point xerj autoindex at an unzipped Confluence HTML space export. A captured run kept page titles, nested page bodies, and the attachment tree.
• How do I search a Slack JSON export? — Point xerj autoindex at an unzipped Slack export. A captured run found a threaded reply, kept ts, thread_ts and user, and round-tripped Japanese text.
• How do I search BVH motion-capture files? — xerj autoindex detects the bvh family and extracts frames, frame_time_s, duration_s, joint_count and a joints keyword array. The family produces no body field.
• How do I search gzipped logs without decompressing the whole file — including when JSONL sits next to them? — XERJ autoindex decompresses a .gz log once at index time. In our capture the plain and gzip copies returned identical counts, identical hits and one date field.
• How do I search Kubernetes pod logs? — Collect the pod logs to disk first, then index the folder. A captured run answered 1 phrase query across 3 pods and returned 18 hits, 6 per pod.
• How do I search through a folder of contracts in .docx? — Index a folder of DOCX contracts with one xerj autoindex command and search their paragraphs. A captured run also shows the decompression guard refusing a file.
• How do I search Unity YAML scenes and the C# that references them as one project? — xerj autoindex detects the unity and unity-meta families, splits a scene into one index per Unity class, and joins an asset to its .meta record by path.
• How do I stop my agent from reading the whole repo into context? — Index the tree once with xerj autoindex, then let the agent query the defs and symbols fields instead of opening files. A dry run first prints every ignore rule.
• How should an agent figure out what's in a messy data folder before searching? — Point xerj autoindex at the folder, then read xerj autoindex map: every dataset it inferred, the fields and types, the time range, and every file it refused.
• I downloaded a bunch of papers as PDFs. How do I search across them? — One xerj autoindex command indexes the folder, then a match_phrase query reads every PDF. A captured run hit all 3 born-digital files and refused a scanned one.
• I have a 2GB SQL dump. How do I find rows mentioning a customer without loading it into Postgres? — xerj autoindex parses a MySQL-style .sql file directly, with no database server, and makes every INSERT value searchable. A captured run indexed 321 documents.
• I have a directory of CSV exports. How do I query them without opening each one in Excel? — Point xerj autoindex at the folder once. A captured run split 3 CSV files across 2 schemas into 2 indices and answered one query across both of them.
• I have a folder of PDFs, Word docs, and markdown. How do I search all of them at once? — Index a folder of PDFs, Word docs and markdown with one xerj autoindex command, then search the contents. A captured 6-file run indexed 51 documents into 2 indices.
• I have a JSON export of Jira issues (or Zendesk tickets). How do I search it like a help-center search? — Index the exported ticket JSON on disk. A captured run returned 36 description hits with per-ticket locators and an exact status count of 30 per status.
• I indexed a folder of email and PDFs. How do I actually read what it found? — xerj brain indexes the folder and the console opens any record: an email with its attachments, a PDF page, and the files it links to. Extracted text only.
• I need to find a function I wrote in a notebook months ago. How do I search all .ipynb files and the .py next to them? — A captured run indexed 2 notebooks by heading section, with the heading path and starting cell, and the .py beside them, all in one index.
• I saved a help-center as HTML. How do I search it like the real help-center search? — XERJ autoindex indexes a static HTML export from local disk, extracts title and a headings array per page, and fetched 0 non-loopback peers in our capture.
• Is it efficient to index PDF files? — PDF extraction is the slowest family XERJ parses. XERJ runs it in an isolated worker sized from free memory, caps it 5 ways, and refuses image-only files.
• Is there a way to full-text search my browser history file itself? — Copy the Chrome History SQLite file and index it. A captured run made titles and URLs queryable, left the source unchanged, and missed 40 WAL rows.
• My agent needs to look up endpoints in a big OpenAPI spec and the SDK markdown. What's the right way? — Index the spec file itself. A captured run found route names, operation descriptions and operationIds in both the JSON and the YAML serialization.
• What's the best way to search markdown notes and PDFs in the same vault? — A captured XERJ run indexed a vault of markdown notes and returned text from inside a PDF attachment and a DOCX attachment, plus wikilink edges between the notes.
• What's the easiest way to search JSON logs plus some old gzip text logs in the same folder? — xerj autoindex detects JSON arrays, JSONL logs and gzipped logs in one folder, infers the date field and answers exact filters. One run returned 386 ERROR hits.
• Why does my Notion export duplicate titles? — A captured run indexed a Notion Markdown export twice and got the same 8 documents with byte-identical ids. The duplicates come from the export layout.

Folder indexing

• I exported my Gmail with Google Takeout. How do I search that mbox file locally? — Unzip the export, run xerj autoindex on the folder, and search messages, threads and attached PDF pages over HTTP. The mailbox is streamed, and with the default embedder no message text leaves your machine.
• I want to point something at a folder and then ask questions about what's in it. What should I use? — Index the folder with one command, read the map of what XERJ found, then let an agent query it over MCP instead of opening the files one by one.

Hybrid retrieval: architecture

• How do I build RAG without a vector database? — Build a RAG retrieval layer on one XERJ index. One hybrid request fuses BM25 and kNN with Reciprocal Rank Fusion, and no second service runs.

Hybrid retrieval: quality

• How much better is hybrid search than BM25? Measured nDCG numbers — XERJ measured BM25, vectors and hybrid RRF on two public BEIR datasets. Hybrid won both: nDCG@10 0.6993 against 0.6572 on SciFact, with a neural embedder.

Hybrid retrieval: reranking

• How do I rerank search results with a relevance judge? — Add a rerank block to a XERJ search: the top 30 hits go to an external judge and a 0 to 1 probability replaces the score. Order by it; the measured calibration says never threshold it with min_score.

Hybrid retrieval: RRF

• What is Reciprocal Rank Fusion? — Reciprocal Rank Fusion scores a document by 1 divided by k plus its rank in each list. XERJ uses k 60, and this page checks the arithmetic by hand.

Hybrid search

• How does XERJ combine BM25 and kNN? — One hybrid request carries a BM25 sub-query and a kNN sub-query against one index, and Reciprocal Rank Fusion merges their ranked lists into a single result list.

Ingest: object storage

• How do I search files that live in an S3 bucket? — Point xerj autoindex at s3://bucket/prefix. It lists the prefix, streams only the objects whose ETag changed, and indexes them. One LIST request per 1,000 keys.

Object storage

• How do I keep an index current when the files live in an object store? — A bucket has no inotify, so you poll or you subscribe to events. Poll cost is the decision: ListObjectsV2 is a Class A operation, one call per 1,000 keys, every cycle.

Operations: completeness

• My codebase indexer says indexed but I don't see my code. How do I check it actually finished? — Compare 4 numbers: files on disk, the dry-run plan, autoindex status, and the index count. A captured run cut 64 files to a 3-file plan, naming a rule per gap.

Operations: estimation

• How do I estimate folder-indexing time? — XERJ measures a client-side extraction floor during a dry run and can stop at a decision gate with exit 4. On 2,500 PDFs the floor was 2.2 min, the run 157.7 s.

Operations: exclusions

• Why would a folder search miss files that I can see on disk? — XERJ prints one line per ignore rule and stores a refusal reason per file. A capture reconciles 26 files on disk to 1 indexed, 2 refused and 23 excluded.

Operations: exit codes

• One dataset was REFUSED by the server. Did I lose the whole index? — When the server refuses one dataset's mapping, xerj autoindex records that dataset's files as junk, indexes every other dataset, and exits 3 with datasets_refused on the terminal line.
• The indexer exited 3 (or 4). Did it fail? — xerj autoindex exits 0 complete, 3 completed-with-junk, 4 needs-a-decision, 2 usage and 1 on an endpoint or journal failure. Only 4 is asking you a question.
• The server pushed back with HTTP 429. Did xerj autoindex lose the run? — When the node answers a bulk, or items inside one, with HTTP 429 (its memory circuit breaker), xerj autoindex re-sends the rejected items, waits out the moment, and reports bulk_retries on the terminal line. Only 600 s of re-sends that never clear is fatal.
• xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why? — autoindex never opens archives. A .zip, .tgz, .7z, .rar or .xz is marked as junk with the exact extraction command; the run continues. Index the extracted folder.

Operations: freshness

• How do I keep a search index up to date as files change, without re-indexing everything? — Run xerj autoindex --watch --no-graph. It indexes once, then reindexes what the filesystem reports. Measured: 0.00 CPU-seconds per idle minute against 1.7 s and a full corpus re-read for every poll.

Operations: installation

• Is there a single-binary local search engine I can just run? — XERJ runs from one native binary with no container and no JVM. A measured capture records 3 linked libraries and a 1.071 s cold start to the first HTTP 200.

Operations: low-volume logs

• What can I use for low-volume log search? — XERJ indexes plain-text logs on one node with exact error counts and sub-millisecond filters. A measured ladder stops at 16 MB: memory grows about 200x the corpus.

Operations: progress

• How do I read autoindex progress? — XERJ prints xerj-bar, xerj-progress and a final xerj-done line. On a single large file the percentage stays at 0.0%, so read since_progress_s and waiting_on.

Operations: recovery

• The indexer died overnight. Do I have to start over? — XERJ keeps a resume journal in the state directory, so re-running autoindex is idempotent. A capture reached 82,231 documents and matched a clean control run.

Operations: reindex

• Reindexing my files is taking forever. What usually causes that? — A measured XERJ _reindex moved 100,000 documents in 20,836 ms on one node. Read the batches, created and failures fields to tell slow progress from no progress.

Product identity

• What is XERJ? — XERJ is an Apache-2.0 single Rust binary that indexes a local folder with one command and answers search, agent memory and MCP calls from the same process.

RAG: troubleshooting

• How do I improve a basic FAISS RAG pipeline? — Move the chunk store into one XERJ index and fuse BM25 with kNN. Two request shapes that a FAISS user reaches for first both return HTTP 400.

Retrieval behaviour

• Why did my filtered vector search get slower when I added a filter? — A filter takes kNN off the HNSW graph and onto an exact scan whose cost scales with vectors scanned. This page explains when that happens and publishes no timing.

Retrieval: selection

• What's the actual difference between a vector database and full-text search when I have a folder? — Full-text search matches tokens, kNN matches vectors, hybrid fuses both. XERJ serves all three from one index over an indexed folder, so the choice is a clause.

Sharing: read-only guest access

• How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it? — Index the folder on your own machine, then run xerj share: the other person gets a link, a passcode and a read-only search page. The folder is not uploaded.
• How do I let someone outside my network search an index on my laptop without opening a port? — xerj share --tunnel starts your own cloudflared, prints a public guest link, and closes the tunnel and revokes the share on Ctrl-C. What the trade is.
• If I share one index with a link and a passcode, what can that person actually reach on my node? — A share-link guest key reads the shared index and nothing else: no other index, no _cat or _cluster, no writes, no scroll or PIT. Each refusal is a live test.