← ALL POSTS · /BLOG

Cover art: 28 filled squares and 217 small empty rings scattered on grid paper, the filled ones joined by hairlines to a wide block of 454 empty cells, beside 32 yellow cells at the same pitch, then a gate of ink posts around two grey-blue columns and one empty yellow cell at its base
PERFORMANCE · AUTOINDEX + SEARCH · 2026-10-26

HOW YOUR FIELD REPORTS
MADE XERJ AUTOINDEX FASTER.

XERJ ENGINEERING · PUBLISHED 2026-10-26

A NOTE FROM IVANWE GOT THIS ONE WRONG, AND YOU FOUND IT FAST

Before the numbers, I owe you an apology. In rc.78 our agentic swarm shipped a change that made every background merge compress stored documents with zstd at level 19, with long-distance matching. It made autoindex about five times slower: a 200 MB bulk test that ran at about 41,000 documents per second on rc.77 managed 5,000 to 14,000 on rc.78 through rc.99. Separately, autoindex split one large folder into 454 small indices, and every search had to open all of them. That was our mistake, and we are sorry for the hours it cost you.

Your field reports, from people and from AI agents, found it very fast. They told us where to look, with numbers, and once the 5 GB reproduction was filed the merge fix was merged about seven hours later, with most of the fixes in this post merged within a day. Below is everything we changed, and a thank-you to every person and every agent that reported. Thank you.

Ivan N

Autoindex and search are faster and leaner now. Your field reports showed us where XERJ had room to get quicker and lighter: large folders that took a long time to index, command-line searches that took tens of seconds, and memory use on small sandboxes. We followed each report to its cause and improved it.

We measured each change on real data, including a 5 GB folder of 146,199 files, and on the performance gate's 1 GB corpus. There, the index now takes 1.80 bytes on disk for each byte of source it holds, down from 3.50. This post explains each improvement in plain terms: where the time and memory went, what we changed, and what you get. It also thanks, by name, the people and the 217 agent field reports that made it happen.

01 · THE RESULTS

Leaner indexes, lighter memory, fewer indices

These numbers come from the performance gate's 1 GB corpus, comparing the current code with the previous release, rc.99, on the same data, the same machine and the same 15-minute run, and from the measurements behind each change.

3.50x → 1.80x
index size per indexed byte
Bytes on disk for each byte of source indexed, between two gate runs on the same corpus.
−74%
client memory
Peak autoindex client memory, 732 MB against 2,830 MB for rc.99.
≤ 32
datasets per autoindex run
The previous release split one 5 GB folder into 454 indices, and every search opened all of them.

On the same run, autoindex moved 29% more data per second than rc.99, and the search node's peak memory was 5% lower. Both runs stopped at the gate's 15-minute limit before indexing the whole corpus, which is why we compare index size per byte indexed. Three of the memory improvements below, 3, 4 and 5, came straight out of an earlier gate run's logs.

02 · WHAT WE IMPROVED

Six improvements, explained simply

Some background first. XERJ writes new documents into small files on disk called segments. In the background it merges small segments into bigger ones, because searching a few big segments is cheaper than searching many small ones. A memory governor watches how much RAM the process uses and paces writes so it stays within its limit. Most of these improvements live in those three parts.

IMPROVEMENT 1

Merges use a faster compression level

DONE

Where the time went. Background merges compressed stored documents with zstd at level 19, with long-distance matching over a 16 MiB window. That setting searches far back through the data for repeated bytes: the output is very small, and the write takes a long time. One merge spent 94–107 s encoding its stored documents, where the normal merge level does the same work in 2.4–3.3 s. While a merge runs, memory sits near the limit, so the governor paces new writes, and a 200 MB bulk test indexed 5,000 to 14,000 documents per second. Before level 19 was used for merges, the same test ran at about 41,000 documents per second.

What we changed. Background merges now compress at the normal merge level, the same one used when new documents are flushed. An explicit force-merge still uses level 19, so you can still ask for the smallest possible index when you have time for it.

What you get. Of the six, this one matters most for autoindex on large folders. Ingest no longer waits behind long merges, and neither does the agent waiting for its index.

IMPROVEMENT 2

File checks went from 1,095 ms to 1.1 ms

DONE

Where the time went. Before autoindex writes a file, it asks the index whether that file is already there. The question itself is a fast count. But the search handler also ran a second step after any request that came back with an empty page of results, to suggest values you might have meant. A count request always has an empty page, so every check also ran that step, which reads a sample of the whole index. Each write also reset the result cache, so the next check ran it again. On a 230,000-document index each check took about 1,095 ms, and the total grew with the number of files times the size of the index.

What we changed. The suggestion step now runs only when you asked for results, got none, and the match is known to be empty. The check now takes about 1.1 ms, roughly a thousand times quicker. Its answer is the same as before, including deletes and writes that are not yet committed.

What you get. A large folder makes many of these checks. Saving about a second on each one adds up to hours on a big folder.

IMPROVEMENT 3

CLI search reads only what it returns

DONE

Where the time went. xerj search builds one query for every index in a project. It adds clauses for code fields such as a symbol's name and its definitions. An index of plain documents has no such fields. The full-text engine could not plan a query that named a field it did not have, so it scanned the stored documents instead, decoding each one in full. About 85% of what it decoded was embedding vectors and their companion columns, which the CLI does not show. One 20,000-document segment decoded 716 MB to answer a single query.

What we changed. A clause on a field that is not mapped anywhere now simply matches nothing, and the rest of the query runs on the full-text index as normal. When a stored scan does run, it decodes only the columns the response can show: that segment now decodes 82 MB in 0.16 s, down from 716 MB in 1.1 s. A cold CLI search that took 13.3–15.3 s now takes 1.07 s cold and 0.14 s warm.

What you get. Agents search in loops, so every second saved per query is saved many times over in a task. The smaller decode also leaves more memory free on a small sandbox.

IMPROVEMENT 4

Merges budget for their final write

DONE

Where the memory went. Before a merge starts, the governor reserves memory for it from an estimate based on its inputs, and the merge checks in between steps so it can pause. The final step, which writes the merged segment, came after the last check. It parses the merged documents, builds the column buffers and compresses them while the merged data is still in memory, and that work had no reservation of its own. In one gate run the node reached 3,920 MB under a 3,072 MB cap, and on a later run large merges reached 4.4–4.9 GB about five seconds before each one finished.

What we changed. The write step is now estimated and charged to the merge budget before it starts, and if it would cross the hard limit, it waits. Merges are also reserved on their real size on disk, and parsed copies of documents are released as soon as they have been used.

What you get. The memory cap now covers every step of a merge. That is what lets XERJ sit comfortably inside a container or a small VM next to your agent.

IMPROVEMENT 5

Compaction always makes progress

DONE

Where the time went. When memory is tight, a merge yields so that searches can run first. On a busy search node, searches alone could keep memory near the limit. A merge would start in a quiet moment, reach its first check, yield, and be planned again with the same inputs, because it was already as small as a batch can be. In one window the same 49 MB batch yielded 68 times before any merge finished. Meanwhile the index stayed split into many segments, and each extra segment is one more file for every search to open.

What we changed. Background yields now have a bound. After three yields in a row with no merge finishing, a batch of 64 MiB or less runs to completion, so compaction keeps moving. Larger batches still yield, so the memory cap still applies.

What you get. Segments keep getting merged, so search stays quick as an index grows, and the node gets its memory back after each merge.

IMPROVEMENT 6

semantic_text stores its vectors once

DONE

Where the space went. An index with a semantic_text field kept two copies of its embedding vectors, and every merge and force-merge moved both of them.

What we changed. On a test of about 6,000 documents, storing the vectors once made the segment file 91% smaller, 1,787,081 bytes instead of 20,148,835. The whole index went from 3.67x to 1.63x the size of its source. Force-merge went from 38 s to about 1 s, and the first semantic query from 610 ms to 219 ms. The savings are on disk rather than in RAM.

What you get. Disk is often the tightest budget on a laptop or in a sandbox, and a smaller index is also quicker to merge and to copy.

Smaller improvements landed alongside these. A bulk request is now checked against the memory limit before it is parsed. Memory for merge buffers is accounted for by the governor, and writes resume once memory has dropped a little below the limit, so they flow smoothly instead of switching on and off. A cold search runs its freshness check alongside the search, reads compressed stored data through a buffer, and reads only document identities until it knows which hits make the page.

All of these improvements are merged, and the changelog describes each one. Once your XERJ includes them, to get the smaller dataset layout on a folder you already indexed, run xerj autoindex --fresh on it. Install steps are on the install page.

03 · WHY THIS MATTERS TO US

Speed and size are the promise

XERJ promises search for agents that ingests in the same league as Lucene, aims to keep its index smaller on disk than Elasticsearch, and is light enough to run beside the agent on a laptop or in a sandbox. Every improvement in this post serves that promise: ingest that keeps pace, answers in milliseconds, and memory that stays within its cap.

So we measured the bar. On the same 1 GB corpus and the same 4 vCPU, 16 GB runner, Elasticsearch 9.5.5 ingested 9.05 MiB/s into an index 1.42x the size of its source. Lucene 10.5.1 ingested 25.4 MiB/s with stored fields at 0.78x, and 36.0 MiB/s at 0.30x without them.

Gate floor, 1 GB tierMust hold
Ingest≥ 3.0 MB/s, and ≥ 0.5x Elasticsearch (stretch 0.8x)
Index size≤ 1.0x Elasticsearch, force-merged
Node memory≤ 3,072 MB RSS, 0 breaker trips
Search latencyp95 ≤ 1,000 ms, max ≤ 2,000 ms, p95 ≤ 1.5x Elasticsearch
Correctness≥ 90% of queries with hits, ≤ 20 bulk retries
Consistencywithin 20% of the last release

Those floors make up a performance gate with three tiers: a 200 MB smoke run, a 1 GB run and a 5 GB pre-release run, on a fixed corpus seed so every run sees the same data. It runs xerj autoindex with default flags, the way you would, on a fresh node with a fixed memory cap, then times uncached searches through the CLI, the API and MCP. A build meets the gate only when it holds every floor: missing any one fails the run and names the floor it missed. The floors live in the repository as the release performance budgets, so changing one takes a reviewed pull request, and loosening one also needs a linked issue that explains why. That is how we hold releases to the bar we measured, and we post each gate run's results openly on GitHub.

04 · THANK YOU

28 people and 217 agent reports

Performance work starts with someone saying “this could be faster”, with numbers. Your reports told us where to look, and we reproduced them on our own hardware. Here is everyone who helped since rc.78, people first, then every agent that sent a field report. Thank you, each of you.

The people who reported performance and memory

The people whose code, reports and tests shaped this work

Thank you, agents

AI agents sent us 217 field reports since rc.78: 169 through the field-report endpoint on xerj.org and 48 on GitHub, as 42 field-report pull requests and 6 issues and discussions. We counted each API report once by its stored report ID, so a report mirrored into two issues counts once, and each GitHub report once by its pull request or issue; where an agent filed both an issue and a pull request for the same run, we counted one. Our own test reports are not included. Each agent below is named as it named itself, with the report that mattered most to us. Bold marks a performance or memory finding.

On GitHub, 43 named reports, with the people who ran each agent:

Through the xerj.org API, 162 named reports, grouped by the name each agent gave:

And 12 more agent reports sent anonymously: 7 through the API without an agent name, and 5 on GitHub (a report by Hamad Ebrahim, a report by Daniel Lan, a report by Riwut Libinuko, a report by Tom Villani, Ph.D., a report by endjoker). Thank you too, whoever you are.

05 · WHAT'S NEXT

What we are improving next

The changelog lists every change with its pull request.

06 · COME IN

Tell us what to make faster next

If XERJ is faster for you, or if there is a place you would like it quicker, tell us with numbers: what you indexed, how big it was, how much memory the machine had, and what you waited for. A short field report is how most of this work started.