--- title: "Is it efficient to index PDF files?" canonical: "https://xerj.org/answers/is-pdf-indexing-efficient" updated: "2026-08-21" source: "content/answers/is-pdf-indexing-efficient.md" --- # Is it efficient to index PDF files? **TL;DR** — Yes, when the alternative is opening every file by hand, and PDF is still the slowest family XERJ parses. XERJ isolates extraction in a worker process, caps it 5 ways, and parses each file once. In our capture 3 born-digital PDFs produced 9 documents, and the 4th file failed with a reason string. ## Agent prompt ```text Act as a coding agent. Read https://xerj.org/llms.txt, start a XERJ node with --insecure, run xerj autoindex on a PDF folder with --prefix pdfx, then query autoindex-catalog for doc_kind run and report pdf_workers, resource_notes, wall_seconds and the pdf_extraction_reuse counters, and query doc_kind file with status junk to report every PDF that failed and its reason string. ``` ## Commands ### Command 1 Note: Index a PDF folder and record the worker decisions on stderr. ```sh xerj autoindex ./pdf --url http://127.0.0.1:9410 --prefix pdfx --state-dir ./state-pdfx --progress plain --disable-feedback ``` ### Command 2 Note: Read the PDF worker count, wall time and extraction reuse counters. ```sh curl -s -XPOST 'http://127.0.0.1:9410/autoindex-catalog/_search' -H 'content-type: application/json' -d '{"query":{"term":{"doc_kind":"run"}},"size":10,"sort":[{"started":"desc"}]}' ``` ### Command 3 Note: List every PDF the run refused, with its reason string. ```sh curl -s -XPOST 'http://127.0.0.1:9410/autoindex-catalog/_search' -H 'content-type: application/json' -d '{"query":{"bool":{"filter":[{"term":{"doc_kind":"file"}},{"term":{"status":"junk"}}]}},"size":20}' ``` ## What one PDF run recorded The run over 4 PDFs indexed 3 files into 9 documents, refused 1 file, and finished in 0.2 seconds with exit code 3. The exit code is the point. A run that refuses a file ends `reason=completed-with-junk` rather than reporting plain success. ```text xerj-done ok=true exit=3 reason=completed-with-junk wall=0.2s files=3 records=9 datasets=1 junk_files=1 ``` ## Why XERJ chose 1 PDF worker XERJ sizes the PDF worker pool from free memory, not from core count, and says so on stderr. The host had 16 cores, and XERJ still ran 1 PDF worker because each worker reserves 1536 MiB. ```text autoindex: 16 scan threads, 16 index workers, 1 pdf workers, --bulk-mb 8 [cores=16 ram_total_mib=64306 ram_available_mib=9004 safe_zone_mib=965] autoindex: memory safe zone 966 MiB allows 1 PDF worker(s) at 1536 MiB each, not 4 ``` Give the machine more free memory if you want more PDF workers. Core count alone does not raise the number. ## Each PDF is parsed once XERJ keeps the text it extracted in phase A and replays it in phase B, so no PDF is parsed twice in one run. The capture created 3 extraction artifacts, replayed 3 of them with 0 integrity failures, and performed 0 second parses. | counter | value | | --- | --- | | `phase_a_pdf_parser_responses` | 3 | | `artifacts_created` | 3 | | `phase_b_pdf_parses` | 0 | | `replay_verified` | 3 | | `replay_integrity_failures` | 0 | | `peak_retained_or_reserved_bytes` | 33,556,112 | | `artifact_ceiling` | 512 | | `byte_ceiling` | 402,653,184 | ## The caps that bound a PDF Five caps bound PDF extraction, and each one refuses rather than degrades. A refusal is recorded in the catalog with its reason, so nothing is silently half-indexed. | cap | value | | --- | --- | | input file | 512 MiB | | pages | 100,000 | | worker output | 32 MiB | | worker address space, Unix | 1.5 GiB | | default timeout | 120 s | The worker is isolated for crash and resource containment. XERJ states in its own help text that the isolation is not a security sandbox, so treat a hostile PDF as hostile. ## An image-only PDF fails, and says why An image-only PDF carries no extractable text, so XERJ refuses it with `status=junk` and 0 documents. The catalog keeps the file with the reason string, which names the fix. ```text PDF has no extractable text; it may be an image-only scan. Run OCR first, then autoindex the searchable PDF; verify/decrypt the PDF, repair it, or run OCR for image-only input ``` XERJ has no optical character recognition of its own. Run OCR outside XERJ, then index the searchable PDF it produces. ## What this capture did not measure The capture recorded no peak resident memory for the PDF worker itself. The run's peak-memory figure of 447,896 kB is `VmHWM` for the node process only. The capture states that the `autoindex` client and the PDF worker are separate processes outside that figure. Treat the 1536 MiB per-worker reservation as the planning number, because it is what XERJ itself uses to decide the worker count. A measured per-worker high-water mark needs a separate run. ## The cost signal at 2,500 PDFs A separate run pointed `autoindex` at 2,500 PDFs with `--max-minutes 1`. The decision gate measured a client-side extraction floor of 2.2 minutes to 2.3 minutes across 10 workers. The gate then exited 4 and wrote a machine-readable decision request. That floor covers client-side extraction only. Server-side indexing time, embedding time and network time are outside it. The same corpus re-run with `--yes` exited 0, and XERJ embeds with lexical feature hashing by default. ## So is PDF indexing efficient PDF indexing pays off when the alternative is to open every file by hand, and PDF is still the slowest family XERJ parses. Plan for extraction time first, memory second, and a refusal path for image-only input. Run `xerj autoindex --dry-run` on the folder first. Set `--max-minutes` to a duration you accept. Every number here came from 1 single-node XERJ process. ## FAQ ### Is PDF the slowest format XERJ indexes? Yes. PDF text extraction is the most expensive family, which is why XERJ runs it in a separate worker process and caps it 5 ways. ### How many PDF workers does XERJ run? As many as free memory allows at 1536 MiB each. Our 16-core host got 1 worker, because the measured memory safe zone was 966 MiB. ### Does XERJ parse the same PDF twice? No. XERJ keeps the phase A extraction and replays it in phase B. Our run replayed 3 artifacts with 0 integrity failures and 0 second parses. ### What happens to an image-only PDF? XERJ refuses it with `status=junk` and 0 documents, and the run exits 3 with `reason=completed-with-junk`. The catalog keeps the reason string. ### Does XERJ run OCR on a PDF? No. XERJ has no optical character recognition. Run OCR outside XERJ, then index the searchable PDF that the OCR step produces. ### How much memory does one PDF worker use? Our capture did not measure it. XERJ reserves 1536 MiB per worker when it sizes the pool, so use that as the planning number. ### How large a PDF can XERJ read? 512 MiB and 100,000 pages, with 32 MiB of worker output and a 120 s default timeout. A file past a cap is refused, never half-indexed. ### Will XERJ warn me before a long PDF job? Yes. The decision gate measures an extraction floor first. On 2,500 PDFs with `--max-minutes 1` it exited 4 and wrote a decision request. ## Related - [I downloaded a bunch of papers as PDFs. How do I search across them?](/answers/search-all-pdfs-in-a-folder) - [I have a folder of PDFs, Word docs, and markdown. How do I search all of them at once?](/answers/search-file-contents-in-a-folder) - [How do I estimate folder-indexing time?](/answers/estimate-autoindex-time-before-running)