Run XERJ fully offline — air-gapped deployment

Goal: follow this short Linux x86_64-musl procedure: stage one operator-approved release, install it as an unprivileged service, and keep the node on loopback. The base path is lexical and needs no model files; neural embedding is optional.

This recipe is traced to the release workflow, embedding loader, and configuration defaults. It is not a claim that an egress-disabled enclave run was live-tested. Execute the final firewall and operational checks in your own target environment.

Runtime boundary

• The default embedding mode is lexical feature hashing: no model, embedding service, or network connection is required.
• Neural mode is optional. Without embedding.local_model_dir, its first embedding operation downloads config.json, tokenizer.json, and model.safetensors for sentence-transformers/all-MiniLM-L6-v2 from Hugging Face; with that directory set, XERJ reads them locally.
• proxy calls only the configured OpenAI-compatible embedding.default_endpoint. The experimental onnx-experimental backend needs an ONNX-enabled build and explicit local assets; neither is the base release path.

Defaults and boundaries

The bundled Console is three embedded documents — index.html, login.html, and setup.html, served under /_xerj-console/ — and each carries the same three external Google Fonts link elements (two preconnects and one stylesheet; the stylesheet may fetch additional font files), so nine in total. Rewriting that HTML means patching all three. Blocked requests fall back to system fonts.

This procedure does not claim that egress was measured here. If policy requires that boundary, apply an egress-deny rule and inspect firewall/log counters during acceptance.

1. Stage an approved release

Run this on a connected staging machine. Set TAG to the operator-approved vX.Y.Z release tag before running; there is no moving release alias here. This procedure stages the Linux x86_64 musl archive only. The matching .sha256 is a separate per-archive asset.

(
  set -eu

  : "${TAG:?set TAG to an operator-approved vX.Y.Z release tag}"
  if [[ ! "$TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*)?(\+[0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*)?$ ]]; then
    echo "TAG must match vX.Y.Z with optional prerelease/build metadata: $TAG" >&2
    exit 2
  fi
  VERSION="${TAG#v}"
  TARGET="x86_64-unknown-linux-musl"
  EXT="tar.gz"
  STAGE="xerj-${VERSION}-${TARGET}"
  ASSET="${STAGE}.${EXT}"
  BASE="https://github.com/xerj-org/xerj/releases/download/${TAG}"
  OUT="${OUT:-$PWD/xerj-airgap-${VERSION}-${TARGET}}"

  mkdir -p "$OUT/release"
  curl -fL --retry 3 -o "$OUT/release/$ASSET" "$BASE/$ASSET"
  curl -fL --retry 3 -o "$OUT/release/$ASSET.sha256" "$BASE/$ASSET.sha256"

  cd "$OUT/release"
  trap 'rm -rf "$STAGE"' EXIT
  want="$(sha256sum "$ASSET" | cut -d ' ' -f 1)"
  if [[ ! "$want" =~ ^[0-9a-f]{64}$ ]]; then
    echo "could not compute a digest for $ASSET" >&2
    exit 1
  fi
  LC_ALL=C grep -qxF -e "$want  $ASSET" -e "$want *$ASSET" \
    < <(tr -d '\r' < "$ASSET.sha256")

  tar -xzf "$ASSET"
  test -x "$STAGE/xerj"
  "$STAGE/xerj" --version
  echo "staged $ASSET ($want) in $OUT"
)

The checksum proves archive integrity against the digest published beside it in the same release; it is not a signature, provenance statement, or attestation. "$STAGE/xerj" --version executes the staged binary, so the staging host must match the target platform; drop that line, or stage a different target by changing TARGET here and the triple spelled out in the enclave block below. The extracted tree is removed by the trap after the cd on a clean exit — success, set -e, and a failed digest. An interrupted run may or may not leave the tree behind: whether the trap runs to completion depends on how and when the interrupt lands, and the honest summary is that you cannot tell from the outside. So do not use the trap to decide safety. The rule that always holds: only the archive and its .sha256 should cross the airgap, and if the block did not print its staged … line, treat the staging directory as unverified — delete it, or inspect it, before transferring anything. The trap is a convenience for the normal exits, not a guarantee across interrupts.

Transfer $OUT itself and land it in the enclave at /opt/xerj-staging, so the archive sits at /opt/xerj-staging/release/ — the fixed path the enclave blocks read. Then set the same approved TAG and verify before extracting:

(
  set -eu

  : "${TAG:?set TAG to the same operator-approved release tag}"
  VERSION="${TAG#v}"
  STAGE="xerj-${VERSION}-x86_64-unknown-linux-musl"
  ASSET="${STAGE}.tar.gz"

  work="$(mktemp -d)"
  trap 'rm -rf "$work"' EXIT
  cp "/opt/xerj-staging/release/$ASSET" "$work/$ASSET"
  cp "/opt/xerj-staging/release/$ASSET.sha256" "$work/$ASSET.sha256"
  cd "$work"

  want="$(sha256sum "$ASSET" | cut -d ' ' -f 1)"
  if [[ ! "$want" =~ ^[0-9a-f]{64}$ ]]; then
    echo "could not compute a digest for $ASSET" >&2
    exit 1
  fi
  LC_ALL=C grep -qxF -e "$want  $ASSET" -e "$want *$ASSET" \
    < <(tr -d '\r' < "$ASSET.sha256")

  tar -xzf "$ASSET"
  test -x "$STAGE/xerj"

  if ! getent passwd xerj >/dev/null; then
    sudo useradd --system --user-group --home-dir /var/lib/xerj \
      --shell /sbin/nologin xerj
  fi
  sudo install -d -m 0755 /opt/xerj/bin /etc/xerj
  sudo install -d -o xerj -g xerj -m 0750 /var/lib/xerj
  sudo install -m 0755 "$STAGE/xerj" /opt/xerj/bin/xerj
  echo "installed $ASSET ($want)"
)

The archive is copied once, then hashed and extracted from that copy. Earlier versions opened it twice at the staging path — once by sha256sum, once by tar — and anyone who can write there swaps the file between the two opens. Everything runs inside the subshell under set -eu, including the TAG guard. Bash does not terminate an interactive shell when a ${var:?} expansion fails, so with the guard outside the block a paste printed its message and carried on with an empty version — and an attacker who can write the staging directory need only leave a file of that name beside the approved one. A subshell is not interactive, so the guard ends the block. The work directory must be writable by you and renameable by nobody else. The default sticky /tmp is exactly right; if yours is not, install -d -m 0700 "$HOME/xerj-verify" and point TMPDIR there. This page once advised a root-owned directory: root ownership was never the problem (/tmp is root-owned), a directory you cannot write is — mktemp then fails, and with that failure unchecked the advice was itself the trigger.

2. Configure the base lexical node

The service account and directories were created by the block above, alongside the install; the ownership shown there is an example, so use the equivalent account-management command for your target image. No model directory is needed for lexical mode.

Create /etc/xerj/xerj.toml with the complete base configuration. Explicit lexical mode and an empty endpoint keep this path model-free:

[server]
bind_address = "127.0.0.1"
data_dir = "/var/lib/xerj"
es_compat_port = 9200

[auth]
enabled = true

[embedding]
mode = "lexical"
default_endpoint = ""
default_model = ""

sudo -u xerj /opt/xerj/bin/xerj --config /etc/xerj/xerj.toml

Start it as xerj. On first start the authenticated admin key is written to /var/lib/xerj/admin.key. Keep the listener on loopback; a network-facing listener needs the production TLS/auth procedure. This command runs in the foreground and stops at logout, so it is a first-start check rather than a deployment — section 4 needs the node reachable from a shell, so use a second terminal or the unit file in Operations — pointing its ExecStart at /opt/xerj/bin/xerj, where the block above installs, since that unit is written for the one-line installer's /usr/local/bin/xerj.

3. Optional neural model

Only when neural semantics are needed, stage the three loader files on the connected machine:

(
  set -eu

  : "${OUT:?set OUT to the staging directory section 1 printed}"
  MODEL="$OUT/model/all-MiniLM-L6-v2"
  mkdir -p "$MODEL"
  for FILE in config.json tokenizer.json model.safetensors; do
    curl -fL --retry 3 -o "$MODEL/$FILE" \
      "https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/$FILE"
  done
  cd "$MODEL"
  sha256sum config.json tokenizer.json model.safetensors > ../model.sha256
  echo "staged all-MiniLM-L6-v2 in $MODEL"
)

Section 1 runs in a subshell, so OUT does not survive it — set it here to the path that block printed. Transfer $OUT/model with the release, in the same /opt/xerj-staging directory. Verify it inside the enclave before installing anything — the same fail-closed rule as section 1, and the reason the check is a command here rather than a sentence:

(
  set -eu

  mwork="$(mktemp -d)"
  trap 'rm -rf "$mwork"' EXIT
  cp /opt/xerj-staging/model/all-MiniLM-L6-v2/{config.json,tokenizer.json,model.safetensors} \
    "$mwork/"
  cp /opt/xerj-staging/model/model.sha256 "$mwork/model.sha256"
  cd "$mwork"

  for f in config.json tokenizer.json model.safetensors; do
    want="$(sha256sum "$f" | cut -d ' ' -f 1)"
    if [[ ! "$want" =~ ^[0-9a-f]{64}$ ]]; then
      echo "could not compute a digest for $f" >&2
      exit 1
    fi
    LC_ALL=C grep -qxF -e "$want  $f" -e "$want *$f" \
      < <(tr -d '\r' < model.sha256)
  done

  sudo install -d -o xerj -g xerj -m 0750 /opt/xerj/models/all-MiniLM-L6-v2
  sudo install -o xerj -g xerj -m 0640 \
    config.json tokenizer.json model.safetensors \
    /opt/xerj/models/all-MiniLM-L6-v2/
  echo "installed all-MiniLM-L6-v2"
)

model.sha256 names all three files, so a transfer that dropped or truncated one fails this check rather than installing a partial model. As in section 1, the block that verifies is the block that installs, so a failed check leaves nothing behind for a later paste to pick up. Replace only the embedding block with:

[embedding]
mode = "neural"
local_model_dir = "/opt/xerj/models/all-MiniLM-L6-v2"

XERJ does not independently verify model checksums; the transferred digest is an operator check, not a XERJ signature. Omit this section and all model assets for lexical search.

4. Authenticate local clients and verify

XERJ=/opt/xerj/bin/xerj
DATA=/var/lib/xerj
KEY="$(sudo cat "$DATA/admin.key")"
export XERJ_API_KEY="$KEY"

"$XERJ" --version
curl -fsS -H "Authorization: ApiKey $KEY" \
  http://127.0.0.1:9200/_cluster/health

With the lexical node running, this semantic query uses the local lexical embedder and needs zero model files. Assert that it returns _id 1:

curl -fsS -X PUT "http://127.0.0.1:9200/offline-demo" \
  -H "Authorization: ApiKey $KEY" -H 'Content-Type: application/json' \
  -d '{"mappings":{"properties":{"body":{"type":"semantic_text"}}}}'
curl -fsS -X POST "http://127.0.0.1:9200/offline-demo/_doc/1?refresh=true" \
  -H "Authorization: ApiKey $KEY" -H 'Content-Type: application/json' \
  -d '{"body":"A local lexical node can answer this without a network service."}'
curl -fsS -X POST "http://127.0.0.1:9200/offline-demo/_search" \
  -H "Authorization: ApiKey $KEY" -H 'Content-Type: application/json' \
  -d '{"query":{"semantic":{"field":"body","query":"local lexical"}}}'

Repeat health and query after restarting the same data directory. For the stronger boundary check, run the sequence with host egress disabled and inspect firewall/log counters; this recipe does not claim that test was run here.

For MCP, XERJ_AUTH and --auth are complete Authorization-header values, including the scheme:

XERJ_URL=http://127.0.0.1:9200 XERJ_AUTH="ApiKey $KEY" "$XERJ" mcp
"$XERJ" mcp --url http://127.0.0.1:9200 --auth "ApiKey $KEY"

For xerj autoindex, XERJ_API_KEY and --api-key take the raw key; the client adds the ApiKey scheme:

XERJ_API_KEY="$KEY" "$XERJ" autoindex /path/to/folder --url http://127.0.0.1:9200
"$XERJ" autoindex /path/to/folder --url http://127.0.0.1:9200 --api-key "$KEY"

Existing data directories and local clients

WAL tap is disabled by default, but PUT /_xerj/wal_tap persists a runtime overlay and reapplies it over TOML on restart. Inspect and remove an old overlay before reusing a data directory:

curl -fsS -H "Authorization: ApiKey $KEY" \
  http://127.0.0.1:9200/_xerj/wal_tap
curl -fsS -X DELETE -H "Authorization: ApiKey $KEY" \
  http://127.0.0.1:9200/_xerj/wal_tap

Cluster mode remains disabled unless explicitly enabled and supplied a shared cluster.auth_secret. xerj mcp is a local stdio client to XERJ_URL; xerj autoindex also defaults to the local ES-compatible endpoint. Neither creates an external listener or silently uploads corpus data.

What this recipe does not promise

• Release .sha256 files provide integrity, not signing or attestation.
• XERJ does not independently checksum the Hugging Face model files.
• The Console falls back to system fonts, but its nine external Google Fonts link elements — three in each of index.html, login.html, and setup.html — remain browser egress attempts unless deployment removes or rewrites all three documents; the stylesheet may fetch additional font files.
• This is a traced procedure, not a live disconnected-firewall run or measured browser-egress claim.

Related: production operations, configuration reference, experimental ONNX backend, and metrics privacy posture.