--- title: "Why does xerj autoindex say the server pushed back (429)?" canonical: "https://xerj.org/answers/autoindex-server-back-pressure-429" updated: "2026-09-20" source: "content/answers/autoindex-server-back-pressure-429.md" --- # The server pushed back with HTTP 429. Did xerj autoindex lose the run? **TL;DR** — No. A per-item HTTP 429 is the node shedding load for a moment. `xerj autoindex` lowers its bulk concurrency, re-sends only the items the node rejected, and continues. The terminal line then carries `bulk_retries=N`. The run exits 1 only after 600 seconds of re-sends that the node never takes, and the same command resumes it. ## Agent prompt ```text Act as a coding agent. Read https://xerj.org/llms.txt, run xerj autoindex with --progress plain, and keep reading the stream when a line says server is shedding load or server pushed back (HTTP 429): the run is re-sending the rejected items and is not hung. Parse the xerj-done line; bulk_retries=N means the run fought back-pressure and still landed every record. Report exit 1 with 'kept rejecting' in the error line as a server condition that did not clear, and rerun the same command to resume. ``` ## Commands ### Command 1 Note: A normal run. If the node sheds load for a moment, the run re-sends the rejected items and continues. ```sh xerj autoindex ./corpus --url http://127.0.0.1:9200 --prefix ax --no-graph --state-dir ./state --progress plain ``` ### Command 2 Note: The node is up while it pushes back: back-pressure is a healthy node protecting itself, not an outage. ```sh curl -s 'http://127.0.0.1:9200/_cluster/health' ``` ### Command 3 Note: If the run did exit 1 because the server kept rejecting for longer than its patience, the same command resumes from the last committed operation. ```sh xerj autoindex ./corpus --url http://127.0.0.1:9200 --prefix ax --no-graph --state-dir ./state --progress plain ``` ## What the node is doing XERJ's engine measures its own resident memory. When it crosses a watermark, a parent memory circuit breaker engages and every write is answered with HTTP 429 until memory drops back below the line. In the node log behind this page the breaker engaged 14 times in 27 minutes and released every engagement within 0.1 to 3.1 seconds. That is the breaker working as designed: the node protects itself instead of being killed for memory. A bulk request can be answered two ways while the breaker is engaged. The whole request can come back HTTP 429, or the request can come back HTTP 200 with some of its items marked `status: 429` inside. Both used to end the run: the second on the spot, the first after six transport retries, about 8 s of backoff in all. ## What used to happen A 48,533-file run aborted at 60.4% of its `index` phase, after 5122.5 seconds of work, when one bulk came back with 747 items rejected 429: ```text xerj-progress phase=index basis=bytes pct=60.4 items=30036/47444 bytes=664119401/1099789983 rate=92730.0 eta_s=4698.3 eta_quality=good since_progress_s=0.0 phase_elapsed_s=4855.0 elapsed_s=5120.1 waiting_on=elasticsearch/libs/h3/src/test/resources/org/elasticsearch/h3/bc14r09centers.txt.gz(732B) autoindex: server pushed back (HTTP 429); lowering bulk concurrency 32 → 16 for this run xerj-done ok=false exit=1 reason=aborted wall=5122.5s error: prepared bulk contained 747 rejected items: {"type":"engine_exception","reason":"[parent] real memory circuit breaker tripped: rss=15679MB >= watermark=15564MB (94% of limit=16384MB); writes rejected to prevent an out-of-memory kill","status":429} ``` The breaker released about a second later. The run was gone. The fix for that shape was then run on the same corpus, resuming the generation, and the other shape ended it 1039.6 s in, at 20.8% of what remained: the engine answered a whole bulk HTTP 429, the transport retry gave up after six attempts, and the run aborted with `error: _bulk: HTTP 429 Too Many Requests: {"took":49,"errors":true,"items":[…]}`. Same breaker, same rejection, a different line of the client. ## What happens now When a bulk comes back HTTP 429 as a whole, or comes back with items rejected 429 and every failed item in it is a 429, the run: 1. Lowers its bulk concurrency, once per congestion event, as it always did. 2. Cuts exactly the rejected actions out of the body it sent. The bulk response lists one answer per action in order, so the mapping is positional, and a `delete` action, which has no document line, keeps its place. 3. Waits a backoff that starts at 250 ms and doubles up to 8 s, then re-sends only those actions. 4. Repeats until the node takes them. It gives up 600 seconds after that bulk was first offered, and not before. The run then exits 1 with an error line that says what happened and what to do: ```text error: the server kept rejecting 12 of a prepared bulk's items after 600s of back-pressure re-sends with nothing accepted: {"type":"engine_exception",...}. Nothing from this bulk was journaled applied; rerun the same command once the server condition clears and the run resumes from its last committed operation ``` A bulk the node refuses for its *size* is not back-pressure either. HTTP 413, or one item answered 413 for a request of many actions, means the request is over `limits.max_body_bytes` or `limits.max_actions_per_bulk`. The run cuts that request in two and sends both halves, keeps the smaller size for every later request, and does not lower its concurrency, because nothing is congested. The terminal line then carries `bulk_splits=N`. [The exit-codes page](/answers/autoindex-exit-codes) has the full-corpus run that used to fail on this. A bulk with a 429 next to a different failure, such as a 400 for a record the node could not parse, is not re-sent. That bulk carries a bad record you need to see, and re-sending the good half would hide which one it was. A whole-request 429 is mapped onto the same loop. When its body is a full bulk response, which is what the engine echoes, the items say which actions were accepted and those are not sent again; when the body is a bare `{"error": …}` object, every action was rejected and the whole body goes out again. A 429 whose body cannot be mapped onto the actions at all is still the transport error it always was, because re-sending on a guess could write the wrong records twice. ## What the stream shows While the run waits, stderr carries one line per 30 seconds at most, per waiting bulk: ```text autoindex: server is shedding load — [parent] real memory circuit breaker tripped: rss=76MB >= watermark=60MB (94% of limit=64MB); writes rejected to prevent an out-of-memory kill; re-offering 2 rejected record(s) (waited 0s, giving up after 600s) ``` The `xerj-progress` line's `since_progress_s` climbs during the wait, because nothing is landing. That is the honest reading. The difference from a hang is the line above: a hang prints no cause. The `raising bulk concurrency` line that follows recovery is printed at most once every 10 seconds, naming the whole climb since the last one, plus the step that reaches the ceiling. The capture that motivated this held 117 of those lines for 11 shrinks. ## What the terminal line says ```text xerj-done ok=true exit=0 reason=completed wall=… files=… records=… bulk_retries=3 ``` `bulk_retries` is the number of re-sent bulks. It appears only when it happened, so a run that never met back-pressure prints the line it always did, and one that did cannot print the same line. On the `--no-graph` path the JSON summary also carries `bulk_items_reissued`, the number of items those re-sends carried. ## If it did exit 1 The node stayed above its memory watermark, accepting nothing, for longer than the run's patience. On the `--no-graph` path the end of the stream says so in words, and says how much is left: ```text autoindex: stopped by server back-pressure while applying : N operation(s) are journaled applied, M are not (this one first) — the same command resumes from here once the node accepts writes again xerj-done ok=false exit=1 reason=server-backpressure wall=… ops_applied=N ops_remaining=M error: the server kept rejecting … ``` `reason=server-backpressure` tells this stop apart from `reason=aborted`, which also covers a broken journal or an unreachable node. The exit code stays 1: exit 3 means a finished run with nothing to retry, and a half-applied generation is not finished. This was forced on a real node by starting it with a 64 MiB memory cap, so the breaker was engaged from the first write. The run re-sent the two rejected records for the full 600 s and ended `xerj-done ok=false exit=1 reason=server-backpressure wall=609.1s ops_applied=0 ops_remaining=231`, printing 19 `server is shedding load` notices on the way — one per 30 s of the wait, not one per re-send. The node was then restarted on its default cap and the same command was run again. It printed `resumed and committed` and ended `ok=true exit=3` with the same 1,663 records and 236 catalog documents as a control run. Read the node log: it names the cap it chose (`memory: detected … usable, using a … cap`) and the setting that changes it, `limits.max_process_memory_mb` or `XERJ_MAX_PROCESS_MEMORY_MB`. Then rerun the same command. Nothing from the rejected bulk was journaled as applied, so the run resumes from its last committed operation and sends that bulk again. One cause is known and is not transient. After a large ingest into many indices the node's resident memory can stay pinned above the watermark: after the run behind this page the node still held 14.8 GB of anonymous memory for 1.2 GB of data on disk, unchanged 2.5 hours after the last write, and every write was 429 until the node was restarted ([#950](https://github.com/xerj-org/xerj/issues/950)). No client-side wait fixes that. Restart the node, or raise the cap, then rerun. ## FAQ ### Why did xerj autoindex say the server pushed back with HTTP 429? The node answered a bulk, or some items inside one, with HTTP 429. Its memory circuit breaker does that for the moments its resident memory sits above a watermark. `xerj autoindex` lowers its bulk concurrency, re-sends only the rejected items after a short backoff, and carries on. ### Is a 429 from the server fatal? Not on its own. Only the items the node rejected are re-sent, after a backoff, for up to 600 s measured from the moment that bulk was first offered. Only then does the run exit 1, and it says so in words: the server kept rejecting after the re-sends, nothing from that bulk was journaled, rerun to resume. ### What does bulk_retries on the xerj-done line mean? How many bulks the run had to re-send because the server answered items with 429. It appears only when it happened. `bulk_retries=3` with `ok=true` means the run fought back-pressure three times and every record still landed. ### Are records written twice when a bulk is re-sent? No. The bulk response lists an answer per action in order, so the run cuts exactly the rejected actions out of the body it sent and re-issues those. Accepted items are not sent again. ### Why does the progress percentage stop moving during back-pressure? Because nothing is landing while the client waits out the backoff. `since_progress_s` climbs honestly and a `server is shedding load` line on stderr names the cause, how long the wait has run and when the run gives up. That is different from a hang, where no such line appears. ### My run exited 1 with 'kept rejecting'. What now? The node stayed above its memory watermark for the whole 600 s a bulk waits. On the `--no-graph` path the terminal line reads `reason=server-backpressure` and carries `ops_applied` and `ops_remaining`. Check the node log for its memory cap, raise `limits.max_process_memory_mb` or give the machine more memory, then rerun the same command: the journal resumes from the last committed operation. If the node log shows resident memory pinned at the watermark long after ingest stopped, that is issue #950 and only a restart of the node clears it. ### Is an HTTP 413 from the node back-pressure? No. A 413 means the request was too large for `limits.max_body_bytes` or `limits.max_actions_per_bulk`. The run cuts the request in two, sends both halves, and reports `bulk_splits=N`. Its concurrency does not drop. ### Does a 429 on a mapping request count the same way? No. A 429 on create-index or put-mapping is the endpoint saying it is busy, not a per-item rejection inside a bulk, and it still aborts the run with exit 1 after the transport retries. Only HTTP 400 there is a dataset refusal. ## Evidence - Before this change, a 48,533-file run aborted at 60.4% of its index phase after 5122.5 s when one bulk came back with 747 items rejected with status 429 by the engine's real memory circuit breaker, and the terminal line read xerj-done ok=false exit=1 reason=aborted wall=5122.5s. — `benchmarks/autoindex-resilience/before-944.full-corpus.stderr.txt` - The node's log for the same run shows the memory circuit breaker engaged 14 times in 27 minutes and released every engagement within 0.1 to 3.1 seconds, at a 16 GiB automatic memory cap on a 119.2 GiB machine. — `benchmarks/autoindex-resilience/before-944.node.governor.txt` - The resumed run, with the per-item fix in place, aborted 1039.6 s in at 20.8% of the remaining index phase with error: _bulk: HTTP 429 Too Many Requests, after the transport retry gave up on a whole-request 429 following six attempts and about 8 s of backoff. — `benchmarks/autoindex-resilience/before-944.whole-request-429.stderr.txt` - After that run the node still held 14.8 GB of anonymous memory for 1.2 GB of data on disk 2.5 hours after the last write, its breaker stayed engaged, and every write was answered 429 until the node was restarted. — [https://github.com/xerj-org/xerj/issues/950](https://github.com/xerj-org/xerj/issues/950) - Only a bulk whose every failed item is a 429 is re-sent; the rejected actions are cut out of the sent body by position and re-issued after a backoff that starts at 250 ms and doubles to 8 s; the client gives up 600 s after that bulk was first offered. — `engine/crates/xerj-autoindex/src/esclient.rs` - The 'raising bulk concurrency' line is printed at most once every 10 s, plus the step that reaches the ceiling; the capture that motivated this holds 117 such lines for 11 shrinks. — `engine/crates/xerj-autoindex/src/esclient.rs` - On a real node started with a 64 MiB memory cap, a --no-graph run ended xerj-done ok=false exit=1 reason=server-backpressure wall=609.1s ops_applied=0 ops_remaining=231 after 600 s of re-sends, and the same command on the restarted node committed the generation with 1,663 records, the same as a control run. — `benchmarks/autoindex-resilience/limits-real-node.txt` - A bulk request the node refuses as too large, with HTTP 413 or with one item answered 413 for a request of several actions, is cut in two and re-sent without lowering the bulk concurrency; on real nodes with max_actions_per_bulk = 64 and max_body_bytes = 98304 each run ended ok=true exit=3 records=1663 bulk_splits=1 with 236 catalog documents, the same as a control run. — `benchmarks/autoindex-resilience/limits-real-node.txt` ## Related - [The indexer exited 3 (or 4). Did it fail?](/answers/autoindex-exit-codes) - [How do I read autoindex progress?](/answers/read-autoindex-progress) - [The indexer died overnight. Do I have to start over?](/answers/resume-interrupted-autoindex-run) - [One dataset was REFUSED by the server. Did I lose the whole index?](/answers/autoindex-dataset-refused-by-server)