RECIPE · 11

Index a folder of PDFs, Word docs & web pages

Goal: point an AI agent at a real, messy shared drive — an HR handbook in PDF, an on-call runbook in Word, a product FAQ in HTML, architecture notes in Markdown, meeting notes in plain text — and let it answer questions with cited passages, regardless of format.

The obvious shell-only approach is to grep/rg the files. It hits a hard ceiling almost immediately: it cannot read a PDF or a .docx (they are compressed binary containers, so the answer is simply invisible), and to actually quote an answer buried in a large document it must slurp the whole file into the model's context window. (A fair grep of the question's own terms does find differently-phrased plaintext answers — so we do not claim a meaning-understanding edge; see the honest scorecard below.)

With XERJ, one xerj-index pass walks the tree, extracts text from every format, chunks it, auto-embeds each chunk with the built-in embedder, and bulk-indexes into a single docfolder index. The agent then searches it three ways — lexical, semantic, and hybrid — and gets back ranked, cited passages.

Everything on this page was measured live. The numbers come from a real run of the indexer + comparison harness in demo/usecases/doc-index against a live XERJ node — a fictional-company corpus ("Northwind Robotics") of 29 files across PDF/DOCX/HTML/MD/TXT (including four realistically large ≥60 KB documents), indexed into 420 chunks, scored over a 22-question ground-truth set. Nothing here is hand-tuned or aspirational; the raw results.json and SCORECARD.md ship next to the scripts so you can reproduce it. The grep baseline is run fairly — it searches the union of each query's keywords and the salient tokens of the question itself.

The problem, made concrete: grep can't read a PDF

The HR handbook is a real PDF. Ask it "how many weeks of paid parental leave do employees receive?" and grep the salient terms — rg returns nothing, because the file is a FlateDecode-compressed binary stream, not text:

$ file corpus/hr/handbook.pdf
corpus/hr/handbook.pdf: PDF document, version 1.7, 5 page(s)

$ head -c 96 corpus/hr/handbook.pdf | cat -v
%PDF-1.7
%M-CM-$M-CM-<M-CM-6M-CM-^_
2 0 obj
<</Length 3 0 R/Filter/FlateDecode>>
stream
x^]XMM-KM-dF...          # compressed bytes — no readable words

$ rg -i -F -e 'parental leave' -e 'paid leave' corpus/
   # (exit 1, no matches — rg detects the NUL bytes and skips the file)

The answer is in there — it is just locked inside the binary. Run a proper extractor and you can see it:

$ pdftotext -layout corpus/hr/handbook.pdf - | grep -i 'parental leave'
Parental leave
Paid parental leave is 16 weeks for eligible full-time staff.

That extraction step is exactly what xerj-index does for every file — and then it chunks, embeds, and indexes the result so you can search it. A naive grep agent is structurally blind to it.

1. Create the `docfolder` index

Two fields carry the document text. body is mapped as semantic_text — XERJ auto-embeds it on ingest for meaning-aware retrieval — and body_text is a plain text field holding the same chunk for BM25 lexical / match. The rest are cheap keyword/metadata fields so you can filter by folder or format and cite the source path.

curl -X PUT localhost:9209/docfolder -H 'Content-Type: application/json' -d '{
  "mappings": {
    "properties": {
      "path":      { "type": "keyword" },
      "dir":       { "type": "keyword" },
      "format":    { "type": "keyword" },
      "title":     { "type": "text" },
      "chunk_id":  { "type": "integer" },
      "body":      { "type": "semantic_text" },
      "body_text": { "type": "text" }
    }
  }
}'

Port note. This walkthrough talks to a node on :9209 — the port the demo harness uses so it never collides with anything else on the box. XERJ's default ES-compat port is :9200; use whichever you started the node on (the scripts honour XERJ_PORT).

2. Index the folder in one pass

The xerj-index.mjs indexer (pure Node, no npm dependencies) walks the folder recursively, extracts text per format, chunks it to ~800 characters with ~100 characters of sentence-boundary overlap, and bulk-indexes each chunk. Extraction uses the right tool for each format:

.pdf → pdftotext -layout (poppler-utils)
.docx → soffice --headless --convert-to txt (LibreOffice), falling back to unzipping word/document.xml and stripping tags
.html → strip <script>/<style>/tags to text (pure Node)
.md / .txt → read as text (light Markdown de-syntaxing)

$ XERJ_PORT=9209 node xerj-index.mjs --recreate
[index] created 'docfolder' with SPEC mappings
[ok] corpus/hr/handbook.pdf (pdf) → 4 chunks
[ok] corpus/engineering/oncall-runbook.docx (docx) → 2 chunks
[ok] corpus/product/faq.html (html) → 1 chunks
...
===== xerj-index summary =====
files walked      : 29
chunks produced   : 420
chunks indexed    : 420
chunks errored    : 0
files skipped     : 0
elapsed ms        : 1339

Measured result: 29 files → 420 chunks, 0 errored, 0 skipped, in 1339 ms. Confirm the index is populated:

$ curl -s localhost:9209/docfolder/_count
{"count":420,"_shards":{"total":1,"successful":1,"failed":0}}

A single file lands as several rows (one per chunk), each with its own path, format, chunk_id, and an auto-embedded body. That is what makes retrieval passage-level: you get the paragraph that answers the question, not the whole 5-page PDF.

3. Search it three ways

Lexical — `match` on `body_text`

Plain BM25 over the extracted text. Good when the query words are literally in the doc:

curl -s localhost:9209/docfolder/_search -H 'Content-Type: application/json' -d '{
  "size": 5,
  "query": { "match": { "body_text": "default API rate limit" } }
}'

{
  "hits": { "hits": [
    { "_index": "docfolder", "_source": {
        "path": "corpus/engineering/architecture/data-pipeline.md",
        "format": "md", "title": "Data Pipeline Notes",
        "body_text": "... The default API rate limit is 1000 requests per minute per client key. ..."
    } }
  ] }
}
// _score + body_vector trimmed for brevity — top hit shown

Semantic — `semantic` on `body`

The user rarely uses the doc's exact words. Ask "what is Northwind's policy on remote work versus coming into the office?" — the facilities note never says "remote work," it says staff "may work from home up to three days a week." The semantic query retrieves it by embedding (word/sub-word) overlap in one shot, without your having to guess synonyms. Be honest about the scope, though: a diligent grep of the question's own salient terms ("work", "office") lands the same line — on differently-phrased plaintext the two approaches tie (see the robustness row below). The convenience is single-query retrieval, not a capability grep lacks:

curl -s localhost:9209/docfolder/_search -H 'Content-Type: application/json' -d '{
  "size": 5,
  "query": { "semantic": { "field": "body",
             "query": "policy on remote work versus coming into the office", "k": 5 } }
}'

{
  "hits": { "hits": [
    { "_index": "docfolder", "_source": {
        "path": "corpus/operations/facilities.md",
        "format": "md", "title": "Facilities and Ways of Working",
        "body_text": "... Northwind runs a distributed-first culture, and staff may work
                       from home up to three days a week. Teams come into the office on the
                       remaining days for planning, workshops, and hands-on time ..."
    } }
  ] }
}

Hybrid — fuse both with RRF

The default the harness scores with: fan out a match on body_text and a semantic query on body, then fuse the two rankings with Reciprocal Rank Fusion. This is what answers the parental-leave question that grep could not touch:

curl -s localhost:9209/docfolder/_search -H 'Content-Type: application/json' -d '{
  "size": 5,
  "query": {
    "hybrid": {
      "queries": [
        { "query": { "match": { "body_text": "weeks of paid parental leave" } } },
        { "query": { "semantic": { "field": "body",
                     "query": "weeks of paid parental leave", "k": 5 } } }
      ],
      "fusion": "rrf"
    }
  }
}'

Measured response — the answer chunk from the binary PDF ranks #1, and every hit carries its source path so the agent can cite it (query latency for this one was 116.09 ms):

{
  "hits": { "hits": [
    { "_index": "docfolder", "_source": {
        "path": "corpus/hr/handbook.pdf", "format": "pdf", "title": "handbook",
        "body_text": "... Parental leave. Paid parental leave is 16 weeks for eligible
                       full-time staff. Parental leave is available to all full-time
                       employees after one year of continuous service ..."
    } },
    { "_source": { "path": "corpus/hr/benefits/leave-policy.docx",     "format": "docx" } },
    { "_source": { "path": "corpus/hr/employee-handbook.html",         "format": "html" } }
  ] }
}
// 5 chunks returned across 3 files (top 3 shown); RRF _score + body_vector trimmed

The measured scorecard: XERJ vs. a fair shell-only grep

The harness runs both approaches over the same 22 ground-truth questions. A query "hits" if any returned passage contains the known answer substring. The grep baseline is fair: it searches the union of each query's curated keywords and the salient tokens of the question itself, so it is never rigged to dodge the answer's own line. The questions are split into four honest buckets:

binary_only (7) — the answer lives only inside a PDF or DOCX. XERJ extracted it; grep is structurally blind. This is the headline — a capability grep lacks.
large_literal (4) — the answer is a literal string buried inside a realistically large (≥60 KB) document. A fair grep finds the line, but must load the whole file to quote/verify it; XERJ returns one ranked passage. This is where the context-efficiency win shows up honestly.
robustness (5) — the answer is phrased differently than the question, in a plaintext file grep can read. A fair grep of the question's own terms matches the same line XERJ's lexical embedder finds, so this is expected to tie — kept to show single-query convenience, not a meaning win.
literal (6) — the answer is a literal substring in a small plaintext file. Both approaches should find these; the honest control that checks XERJ isn't just inflating the easy cases.

Metric                                   XERJ            Baseline (grep)
-------------------------------------    ------------    ---------------
Overall coverage                         21/22 (95.5%)   14/22 (63.6%)
binary_only  (PDF/DOCX-only)  HEADLINE   6/7             0/7
large_literal (buried in ≥60 KB doc)     4/4             3/4
robustness   (differently phrased)       5/5             5/5   ← tie
literal      (plaintext substring)       6/6             6/6
Query latency p50 / mean / max (ms)      61.23 / 64.9 / 116.09
Index build time (ms)                    1339

Coverage by the format the answer lives in — the binary formats are the whole story:

Format   Queries   XERJ            Baseline
------   -------   -----------     ----------
docx     3         2/3 (66.7%)     0/3 (0%)
pdf      5         5/5 (100%)      0/5 (0%)
html     5         5/5 (100%)      5/5 (100%)
md       7         7/7 (100%)      7/7 (100%)
txt      2         2/2 (100%)      2/2 (100%)

The decisive gap is exactly where you'd predict, and only there: on answers locked in PDF/DOCX, grep gets 0/7 and XERJ gets 6/7 (it misses one — see the honest miss below). On the plaintext buckets a fair grep keeps up — robustness ties 5/5 and the literal control ties 6/6 — so XERJ is not winning the cases grep can actually read. What XERJ adds on those is single-query, ranked retrieval and, on large documents, far less context.

Context efficiency — real, but scale-dependent (not flat)

To quote or verify an answer, the grep baseline must open the answer-containing file the hit lives in; XERJ returns just the ranked passage(s). The baseline is charged only that single answer file (statSync(answer_path)), and 0 bytes on any query it cannot answer — never the false-positive files its broad terms also matched. Measured that way, the win is genuine but scale-dependent: large on big documents, and it INVERTS on tiny ones:

View                                     Baseline B   XERJ B    Ratio
--------------------------------------   ----------   -------   --------
large_literal (returned passages)          192,782    11,180    17.24×   ← THE REAL WIN
large_literal (single best passage)        192,782     2,250    85.68×
literal (returned passages)                  4,741    20,965     0.23×   ← INVERTS on tiny files
literal (single best passage)                4,741     3,930     1.21×
answerable (all 14 baseline answers)       202,063    50,750     3.98×
naive overall (all 22 queries)             202,063    79,872     2.53×   ← NOT a claim*

* The naive whole-corpus ratio flatters the blind baseline — it opens 0 bytes on every query it cannot answer — so we do not lean on it. The headline rests on the large_literal ratio (17.24×); on the small-file literal set the same metric drops to 0.23×, because a handful of returned passages can meet or exceed a tiny whole file. Side by side — large_literal 17.24× vs. literal 0.23× — is the scale-dependence, measured rather than asserted; that is why the corpus deliberately includes four ≥60 KB documents, to show the win where it is real. (Diagnostic only, used in no ratio or claim: had the baseline instead been charged every file its broad terms matched — false positives included — the total would be 4,218,746 bytes.)

Honest caveats

This repo has a standing honesty bar. Read these before you rely on any of the above.

The built-in embedder is lexical, not neural. XERJ's zero-config embedder is a deterministic feature-hashing model (384 dimensions, L2-normalised → cosine). It captures word / sub-word overlap, not deep meaning. That is exactly why the robustness bucket is an honest 5/5 tie: a fair grep of the question's own terms matches the same lines XERJ's lexical embedder finds. We make no "beats grep on meaning" claim. For production-grade semantics, point the semantic_text field at an external OpenAI-compatible /v1/embeddings endpoint; the ingest and query paths are identical.
kNN is exact brute-force at query time. The semantic sub-query scans every chunk vector. That's fine (and gives exact results) at this corpus size — the measured max query latency was 116.09 ms over 420 chunks — but latency grows with the number of vectors scanned.
Extraction quality is only as good as the extractor. pdftotext/soffice handle digital PDFs and Word files well, but a scanned PDF (an image of text) yields nothing without an OCR step, which this recipe does not include. Garbled extraction = garbled index. The indexer logs and skips files it cannot read rather than crashing.
Context efficiency is real, but scale-dependent — not free on tiny files. The baseline is charged only the single answer-containing file it must open (and 0 bytes when it cannot answer at all). On the large_literal set XERJ returns 17.24× less context than the whole answer files a fair grep must open (85.68× on a single best passage). But on the small-file literal set the same metric INVERTS to 0.23× — a few returned passages can exceed a tiny whole file — and the naive whole-corpus ratio (2.53×) flatters the blind baseline, so we headline the measured large-document number, not the naive one. The context win appears as documents grow, and honestly disappears on tiny ones.
The one miss (q05). "What is the acknowledgement SLA for a Sev-1 incident?" — the answer, "within 5 minutes," lives in oncall-runbook.docx. It was extracted and indexed (a direct match_phrase on the live index returns exactly that chunk), but for this phrasing hybrid ranking pushes it out of the top-5: the answer chunk lacks the word "SLA," and larger incident-heavy docs outrank it. So binary_only is honestly XERJ 6/7 — still strictly > the baseline's 0/7. We did not change the retrieval strategy to force this to pass.
The baseline could be upgraded. A script agent could shell out to pdftotext/soffice and close the binary gap. But it would still have no ranking, no per-chunk retrieval, no semantic layer, and would keep paying the whole-file context cost. XERJ's win is not "it can read PDFs" alone — it's format-agnostic extraction plus chunked, ranked, context-aware retrieval with a real (if shallow) semantic layer.

Reproduce it yourself

The whole thing — corpus generator, indexer, grep baseline, and comparison harness — lives in demo/usecases/doc-index. Prereqs: Node 18+ (tested on v24), pdftotext (poppler-utils), soffice/LibreOffice, and rg (ripgrep). Then:

# 1. Start a throwaway XERJ (ES-compat wire; default port :9200)
xerj --insecure --data-dir ./docfolder-data

# 2. Point the scripts at your node's port (the demo assumes :9209)
export XERJ_PORT=9200

# 3. (Optional) regenerate the fictional corpus + ground-truth queries.json
node gen-corpus.mjs

# 4. Extract, chunk, embed and bulk-index the whole folder
node xerj-index.mjs --recreate

# 5. Score XERJ vs the grep baseline → writes results.json + SCORECARD.md
node compare.mjs

Open SCORECARD.md for the per-query table and the honest verdict, or results.json for the machine-readable numbers. The full contract these scripts obey — index mappings, chunking, query schema, scoring, and the gate — is documented in SPEC.md and the deliverable's README.md. Every figure on this page is emitted by that run; nothing is hardcoded.