--- title: "How do I rebuild a reference-coding corpus index from scratch?" canonical: "https://xerj.org/answers/rebuild-reference-corpus-index" updated: "2026-09-20" source: "content/answers/rebuild-reference-corpus-index.md" --- # xc-index.sh --fresh failed. How do I rebuild a corpus index? **TL;DR** — Run `xc-index.sh --fresh`. It builds a replacement beside the existing index, checks that it holds records, switches the state file, and only then deletes the old indices by exact name. A build that fails costs you nothing. Older versions of the script failed on every corpus that had been indexed before. ## Agent prompt ```text Act as a coding agent maintaining a reference-coding corpus. Read https://xerj.org/llms.txt. When xc.py reports an index older than 30 days, or a plain xc-index.sh run says the state directory cannot become generation authority, run tools/xerj-code/scripts/xc-index.sh --fresh. Do not delete the old indices yourself and do not pass --fresh to xerj autoindex directly. Read the script's last lines: it states the record count of the verified replacement, or that the existing index was NOT touched. ``` ## Commands ### Command 1 Note: Index a corpus, or update in place the build an earlier run recorded. ```sh bash tools/xerj-code/scripts/xc-index.sh xerj-storage ``` ### Command 2 Note: Rebuild from scratch: build beside the old index, verify, switch, then retire. ```sh bash tools/xerj-code/scripts/xc-index.sh xerj-storage --fresh ``` ### Command 3 Note: Read the ledger. prefix is the whole namespace; index_prefix is the one verified build. ```sh bash -c 'cat ~/.xerj-code/state/xerj-storage.json' ``` ### Command 4 Note: Count the records yourself. The script already did, and refused to switch on zero. ```sh curl -s 'http://127.0.0.1:9200/xc-xerj-storage*/_count' ``` ### Command 5 Note: Query the rebuilt corpus. xc.py addresses the verified build, not the whole namespace. ```sh env XERJ_URL=http://127.0.0.1:9200 tools/xerj-code/scripts/xc.py xerj-storage 'merge policy segment selection' ``` ## Why the old --fresh failed The script used to forward `--fresh` to `xerj autoindex`. That flag means something narrower than its name suggests. It discards the resume journal. It never removes records from the node. And it is refused once a durable corpus generation exists. So the rebuild failed in both states a previously indexed corpus can be in. A state directory written before the `generation-v1` format cannot be adopted: ```text error: this state directory contains a legacy nonempty plan that cannot become generation authority ``` A state directory written since refuses the flag outright: ```text error: this attempt made no remote mutations. `--fresh` cannot discard committed corpus generation 1 under the same destination ``` Both messages give the same advice: build into a new `--state-dir` and a new `--prefix`, validate, then switch readers. A wrapper that owns the whole `xc-` namespace can do exactly that. Now it does. ## What --fresh does now | Step | What happens | What can go wrong | | --- | --- | --- | | Build | `xerj autoindex` runs with prefix `xc--b` and a state directory of its own. `--fresh` is not forwarded. | Nothing an earlier run left behind can be adopted, or can refuse. | | Verify | autoindex exited 0 or 3, and `_count` for the new prefix is above zero. | A zero count fails the build, even on exit 0. | | Switch | `state/.json` is rewritten by atomic rename. | A reader sees the old build or the new one, never half. | | Retire | The old indices are deleted by exact name, then their catalog documents and state directory. | A crash between switch and retire leaves a duplicate, never a gap. | The old indices are listed before the build starts. The retire step can therefore never include what the run itself created. ## A failed build costs nothing If autoindex aborts, or exits 0 with zero records, the script removes only what that build created. The old indices, the old state file and the old state directory stay exactly as they were, and the script says so: ```text xc-index: the existing index was NOT touched and is still what xc.py serves. ``` There is one case where a failed build is kept: it wrote records, and there is no working index to fall back to. Throwing it away would leave no corpus at all. autoindex can abort in finalisation after every document was written, and it can also stop part-way through a large corpus, for example when the node keeps rejecting writes. The script cannot tell the two apart, so it treats both the same way. It records the build, stores the real exit code in `autoindex_exit` with `salvaged: true`, and prints a warning that coverage is not guaranteed. It never swaps a working index out for a build that failed. A kept build can be partial, so the reader is told. While the state file says `salvaged`, or records an exit code other than 0 or 3, `xc.py` prints this on stderr with every query, and `xc.py --list` marks the corpus `INCOMPLETE`: ```text WARNING: the index for 'xerj-storage' was NOT verified complete (autoindex exit 1, kept unverified). Coverage may be INCOMPLETE: a miss here is not evidence that the code is absent. Re-run xc-index.sh xerj-storage to resume or confirm it. ``` ## A count the node does not answer is not zero Every decision above rests on a record count, and a busy node does not always answer one. A timeout, a 5xx or a refused connection is "did not say". It is not zero. The script asks up to 6 times, 5 seconds apart (`XC_COUNT_TRIES`, `XC_COUNT_PAUSE`). Only a number, or a 404 because no index matches, is believed. When the node never says how many records the existing index holds, the script presumes it is a working index. A failed build is then not kept over it. When the node never says how many records the new build holds, nothing is deleted and nothing is switched. The build's indices and its state directory are kept. If there is a working index it keeps serving, and the next `--fresh` that verifies retires the leftover. If there is none, the build is recorded as unverified, `xc.py` warns as above, and a plain re-run confirms it once the node answers. An earlier draft of this script read "did not say" as zero. One 503 on one request then retired a working 500-record index in favour of a failed build, and another deleted a finished build together with its resume state. Both are reproduced in the test file and fail against that draft. A plain `xc-index.sh ` resumes a kept build. The kept build has its own prefix and state directory, so the re-run passes both back to autoindex, which continues from its journal. When that run exits 0 or 3 the `salvaged` mark is cleared and the warning stops. ## Two prefixes in the state file ```text {"corpus":"xerj-storage","prefix":"xc-xerj-storage","index_prefix":"xc-xerj-storage-b20260918101500", ...} ``` `prefix` is always `xc-`. It matches every build's indices, so anything that globs `xc-*` keeps working across a rebuild. `index_prefix` is the one build that was verified. `xc.py` queries it. During a rebuild the namespace holds two builds, and a query against the whole namespace would return passages twice, or from a half-built index. A state file written before builds existed has no `index_prefix`. `xc.py` falls back to `prefix` for it. ## Siblings are safe `xc-battle-*` also matches the indices of a corpus named `battle-terse`. The script knows which corpus names extend this one, from `corpora/` and from `state/`, and leaves their indices out of every list. Deletes are by exact index name. There is no wildcard delete anywhere in the script. ## A plain re-run updates in place `xc-index.sh ` without `--fresh` re-runs autoindex against the recorded build's prefix and state directory. Additions, edits, deletions and renames reconcile incrementally, and the index prefix does not change. If that run fails with the `generation authority` message, the corpus predates the generation format. Run `--fresh` once. ## What to plan for A rebuild holds the corpus on the node twice until the old build is retired. Plan disk for double. This was not measured. The old state directory of a corpus indexed before builds existed lives under `~/.xerj/autoindex/`, in a folder named by a hash. The script cannot compute that name, so it leaves the folder behind. It is harmless, and you can delete it by hand. ## How this page was checked The two error messages are captures from v1.0.0-rc.74, committed under `benchmarks/autoindex-resilience/`. Local paths in them were shortened; nothing else was changed. The new behaviour is pinned by `tools/xerj-code/tests/test_xc_index_fresh.py`: 81 offline checks against a fake node and a fake `xerj` binary that refuses what the real one refuses. The same test fails against the old script with the first error above. The warning and the `--list` mark are pinned by `tools/xerj-code/tests/test_state_ledger.py` (25 offline checks). Resuming a kept build was checked against the fake binary only: that the re-run passes the same prefix and state directory. autoindex resuming an interrupted generation from its journal is a separate capture, `benchmarks/autoindex-resilience/after-fix.resume-probe.stderr.txt`. The two were not run together end to end. Writing that test found five defects in drafts of the new script. An index listing that never reached its filter, a record count that read as zero when the node put a space after the colon, two rebuilds inside one second retiring the build they had just verified, and the two unanswered-count defects described above. All five are fixed and covered. The unanswered count was exercised against the fake node only, with a 503. It was not provoked on a real node. ## FAQ ### How do I rebuild a reference-coding corpus index from scratch? Run `xc-index.sh --fresh`. It builds a replacement beside the existing index, verifies that it holds records, switches the state file, and only then deletes the old indices by exact name. ### xc-index.sh --fresh failed. How do I rebuild a corpus index? Update the script. Older versions forwarded `--fresh` to `xerj autoindex`, which refuses it once a corpus generation has committed. The current script never forwards it and rebuilds under a new prefix instead. ### Does --fresh delete my working index first? No. The old index stays live until the replacement exited 0 or 3 and its record count is above zero. A build that fails or comes back empty is removed, and the old index and state file are left exactly as they were. ### Why does the state file have both prefix and index_prefix? `prefix` is always `xc-`, the whole namespace, so anything that globs on it keeps working. `index_prefix` is the one verified build, and it is what `xc.py` queries. ### Is xc-index.sh --fresh the same as xerj autoindex --fresh? No. `xerj autoindex --fresh` only discards the resume journal, never removes records, and is refused once a durable generation exists. The script's `--fresh` is a full rebuild under a new prefix and state directory. ### What happens if the first build of a corpus is interrupted? With no working index to fall back to, the script keeps the partial build, records `salvaged: true` and the real exit code in the state file, and warns. `xc.py` then warns on every query that coverage is incomplete. A plain `xc-index.sh ` resumes the run under the same prefix and state directory. ### How much disk does a rebuild need? Room for the corpus twice. Both builds exist on the node from the start of the rebuild until the old one is retired. This was not measured; plan for double. ## Evidence - On v1.0.0-rc.74 a state directory written before the generation-v1 format aborted with exit 1: this state directory contains a legacy nonempty plan that cannot become generation authority. — `benchmarks/autoindex-resilience/fresh-before-rc74.legacy-state.txt` - On v1.0.0-rc.74 xerj autoindex --fresh over a committed generation aborted with exit 1: --fresh cannot discard committed corpus generation 1 under the same destination. — `benchmarks/autoindex-resilience/fresh-before-rc74.committed-generation.txt` - The --fresh contract is pinned by 81 offline checks: a new state directory and prefix, --fresh never forwarded to autoindex, nothing deleted before the replacement's count was read, deletes by exact name, a failed or empty build leaves the old index and state file untouched, a sibling corpus is never touched, an interrupted first build is kept, recorded as salvaged with its real exit code, and resumed by a plain re-run under the same prefix and state directory, and a record count the node does not answer is never read as zero: it cannot get a working index retired or a finished build deleted. — `tools/xerj-code/tests/test_xc_index_fresh.py` - When the ledger records a salvaged build or an autoindex exit other than 0 or 3, xc.py prints a coverage-may-be-INCOMPLETE warning on stderr on every query, keeps --json stdout parseable, and marks the corpus INCOMPLETE in --list. Exit 3 is not reported as incomplete. — `tools/xerj-code/tests/test_state_ledger.py` - xc.py queries index_prefix, the one verified build, and never the bare namespace glob once a corpus has been rebuilt. — `tools/xerj-code/tests/test_state_ledger.py` ## Related - [How do I set up reference coding so the agent retrieves a mechanism before writing code?](/answers/index-reference-library-for-coding-agent) - [The indexer died overnight. Do I have to start over?](/answers/resume-interrupted-autoindex-run) - [The indexer exited 3 (or 4). Did it fail?](/answers/autoindex-exit-codes) - [My codebase indexer says indexed but I don't see my code. How do I check it actually finished?](/answers/check-codebase-index-is-complete)