xc-index.sh --fresh failed. How do I rebuild a corpus index?

TL;DR — Run xc-index.sh <corpus> --fresh. It builds a replacement beside the existing index, checks that it holds records, switches the state file, and only then deletes the old indices by exact name. A build that fails costs you nothing. Older versions of the script failed on every corpus that had been indexed before.

Why the old --fresh failed

The script used to forward --fresh to xerj autoindex. That flag means something narrower than its name suggests. It discards the resume journal. It never removes records from the node. And it is refused once a durable corpus generation exists.

So the rebuild failed in both states a previously indexed corpus can be in.

A state directory written before the generation-v1 format cannot be adopted:

error: this state directory contains a legacy nonempty plan that cannot become generation authority

A state directory written since refuses the flag outright:

error: this attempt made no remote mutations. `--fresh` cannot discard committed corpus generation 1 under the same destination

Both messages give the same advice: build into a new --state-dir and a new --prefix, validate, then switch readers. A wrapper that owns the whole xc-<corpus> namespace can do exactly that. Now it does.

What --fresh does now

The old indices are listed before the build starts. The retire step can therefore never include what the run itself created.

A failed build costs nothing

If autoindex aborts, or exits 0 with zero records, the script removes only what that build created. The old indices, the old state file and the old state directory stay exactly as they were, and the script says so:

xc-index: the existing index was NOT touched and is still what xc.py serves.

There is one case where a failed build is kept: it wrote records, and there is no working index to fall back to. Throwing it away would leave no corpus at all. autoindex can abort in finalisation after every document was written, and it can also stop part-way through a large corpus, for example when the node keeps rejecting writes. The script cannot tell the two apart, so it treats both the same way. It records the build, stores the real exit code in autoindex_exit with salvaged: true, and prints a warning that coverage is not guaranteed. It never swaps a working index out for a build that failed.

A kept build can be partial, so the reader is told. While the state file says salvaged, or records an exit code other than 0 or 3, xc.py prints this on stderr with every query, and xc.py --list marks the corpus INCOMPLETE:

WARNING: the index for 'xerj-storage' was NOT verified complete (autoindex exit 1, kept unverified). Coverage may be INCOMPLETE: a miss here is not evidence that the code is absent. Re-run xc-index.sh xerj-storage to resume or confirm it.

A count the node does not answer is not zero

Every decision above rests on a record count, and a busy node does not always answer one. A timeout, a 5xx or a refused connection is "did not say". It is not zero. The script asks up to 6 times, 5 seconds apart (XC_COUNT_TRIES, XC_COUNT_PAUSE). Only a number, or a 404 because no index matches, is believed.

When the node never says how many records the existing index holds, the script presumes it is a working index. A failed build is then not kept over it.

When the node never says how many records the new build holds, nothing is deleted and nothing is switched. The build's indices and its state directory are kept. If there is a working index it keeps serving, and the next --fresh that verifies retires the leftover. If there is none, the build is recorded as unverified, xc.py warns as above, and a plain re-run confirms it once the node answers.

An earlier draft of this script read "did not say" as zero. One 503 on one request then retired a working 500-record index in favour of a failed build, and another deleted a finished build together with its resume state. Both are reproduced in the test file and fail against that draft.

A plain xc-index.sh <corpus> resumes a kept build. The kept build has its own prefix and state directory, so the re-run passes both back to autoindex, which continues from its journal. When that run exits 0 or 3 the salvaged mark is cleared and the warning stops.

Two prefixes in the state file

{"corpus":"xerj-storage","prefix":"xc-xerj-storage","index_prefix":"xc-xerj-storage-b20260918101500", ...}

prefix is always xc-<corpus>. It matches every build's indices, so anything that globs xc-<corpus>* keeps working across a rebuild.

index_prefix is the one build that was verified. xc.py queries it. During a rebuild the namespace holds two builds, and a query against the whole namespace would return passages twice, or from a half-built index.

A state file written before builds existed has no index_prefix. xc.py falls back to prefix for it.

Siblings are safe

xc-battle-* also matches the indices of a corpus named battle-terse. The script knows which corpus names extend this one, from corpora/ and from state/, and leaves their indices out of every list. Deletes are by exact index name. There is no wildcard delete anywhere in the script.

A plain re-run updates in place

xc-index.sh <corpus> without --fresh re-runs autoindex against the recorded build's prefix and state directory. Additions, edits, deletions and renames reconcile incrementally, and the index prefix does not change.

If that run fails with the generation authority message, the corpus predates the generation format. Run --fresh once.

What to plan for

A rebuild holds the corpus on the node twice until the old build is retired. Plan disk for double. This was not measured.

The old state directory of a corpus indexed before builds existed lives under ~/.xerj/autoindex/, in a folder named by a hash. The script cannot compute that name, so it leaves the folder behind. It is harmless, and you can delete it by hand.

How this page was checked

The two error messages are captures from v1.0.0-rc.74, committed under benchmarks/autoindex-resilience/. Local paths in them were shortened; nothing else was changed.

The new behaviour is pinned by tools/xerj-code/tests/test_xc_index_fresh.py: 81 offline checks against a fake node and a fake xerj binary that refuses what the real one refuses. The same test fails against the old script with the first error above.

The warning and the --list mark are pinned by tools/xerj-code/tests/test_state_ledger.py (25 offline checks). Resuming a kept build was checked against the fake binary only: that the re-run passes the same prefix and state directory. autoindex resuming an interrupted generation from its journal is a separate capture, benchmarks/autoindex-resilience/after-fix.resume-probe.stderr.txt. The two were not run together end to end.

Writing that test found five defects in drafts of the new script. An index listing that never reached its filter, a record count that read as zero when the node put a space after the colon, two rebuilds inside one second retiring the build they had just verified, and the two unanswered-count defects described above. All five are fixed and covered. The unanswered count was exercised against the fake node only, with a 503. It was not provoked on a real node.

FAQ

How do I rebuild a reference-coding corpus index from scratch?

Run xc-index.sh <corpus> --fresh. It builds a replacement beside the existing index, verifies that it holds records, switches the state file, and only then deletes the old indices by exact name.

xc-index.sh --fresh failed. How do I rebuild a corpus index?

Update the script. Older versions forwarded --fresh to xerj autoindex, which refuses it once a corpus generation has committed. The current script never forwards it and rebuilds under a new prefix instead.

Does --fresh delete my working index first?

No. The old index stays live until the replacement exited 0 or 3 and its record count is above zero. A build that fails or comes back empty is removed, and the old index and state file are left exactly as they were.

Why does the state file have both prefix and index_prefix?

prefix is always xc-<corpus>, the whole namespace, so anything that globs on it keeps working. index_prefix is the one verified build, and it is what xc.py queries.

Is xc-index.sh --fresh the same as xerj autoindex --fresh?

No. xerj autoindex --fresh only discards the resume journal, never removes records, and is refused once a durable generation exists. The script's --fresh is a full rebuild under a new prefix and state directory.

What happens if the first build of a corpus is interrupted?

With no working index to fall back to, the script keeps the partial build, records salvaged: true and the real exit code in the state file, and warns. xc.py then warns on every query that coverage is incomplete. A plain xc-index.sh <corpus> resumes the run under the same prefix and state directory.

How much disk does a rebuild need?

Room for the corpus twice. Both builds exist on the node from the start of the rebuild until the old one is retired. This was not measured; plan for double.

Evidence

• On v1.0.0-rc.74 a state directory written before the generation-v1 format aborted with exit 1: this state directory contains a legacy nonempty plan that cannot become generation authority. — benchmarks/autoindex-resilience/fresh-before-rc74.legacy-state.txt
• On v1.0.0-rc.74 xerj autoindex --fresh over a committed generation aborted with exit 1: --fresh cannot discard committed corpus generation 1 under the same destination. — benchmarks/autoindex-resilience/fresh-before-rc74.committed-generation.txt
• The --fresh contract is pinned by 81 offline checks: a new state directory and prefix, --fresh never forwarded to autoindex, nothing deleted before the replacement's count was read, deletes by exact name, a failed or empty build leaves the old index and state file untouched, a sibling corpus is never touched, an interrupted first build is kept, recorded as salvaged with its real exit code, and resumed by a plain re-run under the same prefix and state directory, and a record count the node does not answer is never read as zero: it cannot get a working index retired or a finished build deleted. — tools/xerj-code/tests/test_xc_index_fresh.py
• When the ledger records a salvaged build or an autoindex exit other than 0 or 3, xc.py prints a coverage-may-be-INCOMPLETE warning on stderr on every query, keeps --json stdout parseable, and marks the corpus INCOMPLETE in --list. Exit 3 is not reported as incomplete. — tools/xerj-code/tests/test_state_ledger.py
• xc.py queries index_prefix, the one verified build, and never the bare namespace glob once a corpus has been rebuilt. — tools/xerj-code/tests/test_state_ledger.py

Related

• How do I set up reference coding so the agent retrieves a mechanism before writing code?
• The indexer died overnight. Do I have to start over?
• The indexer exited 3 (or 4). Did it fail?
• My codebase indexer says indexed but I don't see my code. How do I check it actually finished?