--- title: "Why is reindexing my files taking forever?" canonical: "https://xerj.org/answers/reindex-taking-forever" updated: "2026-08-23" source: "content/answers/reindex-taking-forever.md" --- # Reindexing my files is taking forever. What usually causes that? **TL;DR** — A XERJ `_reindex` of a fixed 100,000-document corpus reported `"took": 20836` milliseconds, 100 batches and 0 failures on one node. Read `created` against a `_count` of the destination index to separate slow progress from a stalled call. XERJ is single-node, so `_reindex` is a local copy. ## Agent prompt ```text Act as a coding agent. Read https://xerj.org/llms.txt, start a single-node XERJ, count the source index, POST /_reindex into a fresh destination index, then compare the response took, batches, created and failures fields against a _count of the destination before you report progress. ``` ## Commands ### Command 1 Note: Count the source index before the copy starts. ```sh curl -s http://127.0.0.1:9200/reindex_src/_count ``` ### Command 2 Note: Copy every document from the source index into a fresh destination index. ```sh curl -s -XPOST http://127.0.0.1:9200/_reindex -H 'content-type: application/json' -d '{"source":{"index":"reindex_src"},"dest":{"index":"reindex_dst"}}' ``` ### Command 3 Note: Count the destination index after the call returns. ```sh curl -s http://127.0.0.1:9200/reindex_dst/_count ``` ## Two different jobs are called reindexing A folder re-run and `POST /_reindex` are different calls. A folder re-run of `xerj autoindex` reads the journal in the state directory and skips the files it already finished. A `_reindex` copies documents from one index into another inside the same node. This page measures the second call. The resume article covers the first. ## What one measured reindex looked like A `_reindex` of 100,000 documents into a fresh destination index returned in 20,836 milliseconds. Harness wall-clock time for the same call was 20.893 s, and the source count and the destination count both read 100,000 afterward. The captured response is small, and every field in it is a progress signal. ```json { "took": 20836, "total": 100000, "created": 100000, "updated": 0, "batches": 100, "version_conflicts": 0, "noops": 0, "retries": { "bulk": 0, "search": 0 }, "throttled_millis": 0, "timed_out": false, "failures": [] } ``` Treat this as one run on one shared host at 1-minute load average 2.63, not as a throughput promise. ## Read the fields before you assume a stall `created` is the field that answers "is it moving". A `_reindex` call blocks until it finishes, so the response arrives only at the end. During the call, read progress from the destination index instead. Count the destination index while the copy runs. ```sh curl -s http://127.0.0.1:9200/reindex_dst/_count ``` A count that climbs means the copy still makes progress. A count that stays flat while the node still answers means one batch is still in flight. ## Why the wall time is what it is The measured call moved 100,000 documents in 100 batches, so each batch carried 1,000 documents. Batch count, document size and mapping complexity set the wall time more than any single tuning flag does. Host load matters too. The capture recorded 1-minute load average 2.63 before the call and 2.27 after it. For context, XERJ measured 1.72× the bulk ingest rate of Elasticsearch 8.13.4 on a separate 100,000-document board. The same board publishes 4 read p99 losses under sustained write, including 13.57 ms for XERJ against 3.45 ms for Elasticsearch. Neither result predicts your `_reindex` time. ## Memory grows while a reindex runs XERJ retains heap per indexed document, which is an open engine defect rather than a tuning problem. A long `_reindex` therefore grows the resident set as it proceeds. The project's own instruction is to not plan corpora beyond a few million documents. Watch free memory on the host, not only the response. ## Check size the way that works Two Elasticsearch habits for reading index size fail on XERJ, and the capture keeps both responses. `GET /_cat/indices?format=json&bytes=b` still returns human-formatted values such as `716.1kb`. The metric-filtered `GET /{index}/_stats/store` returns 404. Use the plain form, then read `_all.total.store.size_in_bytes` from the response. ```sh curl -s http://127.0.0.1:9200/reindex_dst/_stats ``` ## What single-node means for a reindex XERJ is single-node, so `_reindex` copies inside one process on one host. There is no data-plane replication and no failover, so a node that stops takes the in-flight batch with it. Re-run the call after a restart and compare the destination `_count` with the source `_count`. ## FAQ ### Reindexing my files is taking forever. What usually causes that? Corpus size, document size, host load and mapping complexity set the wall time. A _reindex call blocks and answers only at the end, so read a _count of the destination before you call it stuck. ### Why is my local search index rebuild so slow? Corpus size, document size, host load and mapping complexity all change the time. The measured run used a fixed 100,000-document corpus at 1-minute load average 2.63. ### What should I measure when folder reindex never finishes? For a _reindex, read took, batches, created, version_conflicts, retries and failures, and count the destination while the call runs. For a folder re-run, read xerj autoindex status, which reports files done, records and the journal state. ### Is PDF extraction the thing that makes reindex hang? Not measured here. This page captures a _reindex of a fixed document corpus, not a per-format split, so it cannot blame PDF extraction. A folder run does bound each PDF with a size cap, a page cap and a worker timeout, and refuses a file past a cap instead of hanging on it. ### How long does a XERJ reindex take? One measured run copied 100,000 documents in 20,836 ms on a single node. That is one run on one shared host, not a rate you can extrapolate. ### Does XERJ reindex across nodes? No. XERJ is single-node, so _reindex is a local copy. There is no cross-node parallelism to add and no failover if the node stops. ### Does reindexing use more memory as it runs? Yes. XERJ has an open defect that retains heap per indexed document, so a large reindex grows resident memory. Do not plan corpora beyond a few million documents. ## Related - [How do I read autoindex progress?](/answers/read-autoindex-progress) - [Migrate from Elasticsearch](/docs/migration-from-es) - [What can I use for low-volume log search?](/answers/cheap-low-volume-log-search) - [The indexer died overnight. Do I have to start over?](/answers/resume-interrupted-autoindex-run)