Is it efficient to index PDF files?
TL;DR — Yes, when the alternative is opening every file by hand, and PDF is still the slowest family XERJ parses. XERJ isolates extraction in a worker process, caps it 5 ways, and parses each file once. In our capture 3 born-digital PDFs produced 9 documents, and the 4th file failed with a reason string.
Act as a coding agent. Read https://xerj.org/llms.txt, start a XERJ node with --insecure, run xerj autoindex on a PDF folder with --prefix pdfx, then query autoindex-catalog for doc_kind run and report pdf_workers, resource_notes, wall_seconds and the pdf_extraction_reuse counters, and query doc_kind file with status junk to report every PDF that failed and its reason string.
xerj autoindex ./pdf --url http://127.0.0.1:9410 --prefix pdfx --state-dir ./state-pdfx --progress plain --disable-feedback
Index a PDF folder and record the worker decisions on stderr.
curl -s -XPOST 'http://127.0.0.1:9410/autoindex-catalog/_search' -H 'content-type: application/json' -d '{"query":{"term":{"doc_kind":"run"}},"size":10,"sort":[{"started":"desc"}]}'
Read the PDF worker count, wall time and extraction reuse counters.
curl -s -XPOST 'http://127.0.0.1:9410/autoindex-catalog/_search' -H 'content-type: application/json' -d '{"query":{"bool":{"filter":[{"term":{"doc_kind":"file"}},{"term":{"status":"junk"}}]}},"size":20}'
List every PDF the run refused, with its reason string.
What one PDF run recorded
The run over 4 PDFs indexed 3 files into 9 documents, refused 1 file, and finished in 0.2 seconds with exit code 3. The exit code is the point. A run that refuses a file ends reason=completed-with-junk rather than reporting plain success.
xerj-done ok=true exit=3 reason=completed-with-junk wall=0.2s files=3 records=9 datasets=1 junk_files=1
Why XERJ chose 1 PDF worker
XERJ sizes the PDF worker pool from free memory, not from core count, and says so on stderr. The host had 16 cores, and XERJ still ran 1 PDF worker because each worker reserves 1536 MiB.
autoindex: 16 scan threads, 16 index workers, 1 pdf workers, --bulk-mb 8 [cores=16 ram_total_mib=64306 ram_available_mib=9004 safe_zone_mib=965]
autoindex: memory safe zone 966 MiB allows 1 PDF worker(s) at 1536 MiB each, not 4
Give the machine more free memory if you want more PDF workers. Core count alone does not raise the number.
Each PDF is parsed once
XERJ keeps the text it extracted in phase A and replays it in phase B, so no PDF is parsed twice in one run. The capture created 3 extraction artifacts, replayed 3 of them with 0 integrity failures, and performed 0 second parses.
| counter | value |
|---|---|
phase_a_pdf_parser_responses | 3 |
artifacts_created | 3 |
phase_b_pdf_parses | 0 |
replay_verified | 3 |
replay_integrity_failures | 0 |
peak_retained_or_reserved_bytes | 33,556,112 |
artifact_ceiling | 512 |
byte_ceiling | 402,653,184 |
The caps that bound a PDF
Five caps bound PDF extraction, and each one refuses rather than degrades. A refusal is recorded in the catalog with its reason, so nothing is silently half-indexed.
| cap | value |
|---|---|
| input file | 512 MiB |
| pages | 100,000 |
| worker output | 32 MiB |
| worker address space, Unix | 1.5 GiB |
| default timeout | 120 s |
The worker is isolated for crash and resource containment. XERJ states in its own help text that the isolation is not a security sandbox, so treat a hostile PDF as hostile.
An image-only PDF fails, and says why
An image-only PDF carries no extractable text, so XERJ refuses it with status=junk and 0 documents. The catalog keeps the file with the reason string, which names the fix.
PDF has no extractable text; it may be an image-only scan. Run OCR first, then autoindex the searchable PDF; verify/decrypt the PDF, repair it, or run OCR for image-only input
XERJ has no optical character recognition of its own. Run OCR outside XERJ, then index the searchable PDF it produces.
What this capture did not measure
The capture recorded no peak resident memory for the PDF worker itself. The run's peak-memory figure of 447,896 kB is VmHWM for the node process only. The capture states that the autoindex client and the PDF worker are separate processes outside that figure.
Treat the 1536 MiB per-worker reservation as the planning number, because it is what XERJ itself uses to decide the worker count. A measured per-worker high-water mark needs a separate run.
The cost signal at 2,500 PDFs
A separate run pointed autoindex at 2,500 PDFs with --max-minutes 1. The decision gate measured a client-side extraction floor of 2.2 minutes to 2.3 minutes across 10 workers. The gate then exited 4 and wrote a machine-readable decision request.
That floor covers client-side extraction only. Server-side indexing time, embedding time and network time are outside it. The same corpus re-run with --yes exited 0, and XERJ embeds with lexical feature hashing by default.
So is PDF indexing efficient
PDF indexing pays off when the alternative is to open every file by hand, and PDF is still the slowest family XERJ parses. Plan for extraction time first, memory second, and a refusal path for image-only input.
Run xerj autoindex --dry-run on the folder first. Set --max-minutes to a duration you accept. Every number here came from 1 single-node XERJ process.
FAQ
Is PDF the slowest format XERJ indexes?
Yes. PDF text extraction is the most expensive family, which is why XERJ runs it in a separate worker process and caps it 5 ways.
How many PDF workers does XERJ run?
As many as free memory allows at 1536 MiB each. Our 16-core host got 1 worker, because the measured memory safe zone was 966 MiB.
Does XERJ parse the same PDF twice?
No. XERJ keeps the phase A extraction and replays it in phase B. Our run replayed 3 artifacts with 0 integrity failures and 0 second parses.
What happens to an image-only PDF?
XERJ refuses it with status=junk and 0 documents, and the run exits 3 with reason=completed-with-junk. The catalog keeps the reason string.
Does XERJ run OCR on a PDF?
No. XERJ has no optical character recognition. Run OCR outside XERJ, then index the searchable PDF that the OCR step produces.
How much memory does one PDF worker use?
Our capture did not measure it. XERJ reserves 1536 MiB per worker when it sizes the pool, so use that as the planning number.
How large a PDF can XERJ read?
512 MiB and 100,000 pages, with 32 MiB of worker output and a 120 s default timeout. A file past a cap is refused, never half-indexed.
Will XERJ warn me before a long PDF job?
Yes. The decision gate measures an extraction floor first. On 2,500 PDFs with --max-minutes 1 it exited 4 and wrote a decision request.