HOME·ANSWERS·HOW DO I SEARCH FILES THAT LIVE IN AN S3 BUCKET?
ANSWERS · INGEST: OBJECT STORAGE

How do I search files that live in an S3 bucket?

AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-09-19 · UPDATED · 2026-09-19

TL;DR — Run xerj autoindex s3://bucket/prefix against a running node. XERJ lists the prefix, streams in only the objects whose ETag or size changed, and indexes them with the same extractors it uses for a folder. Credentials come from the environment. The index stays on local disk.

PASTE THIS TO YOUR AI AGENT — IT DOES THE REST Act as a coding agent. Read https://xerj.org/llms.txt, then index the bucket with xerj autoindex s3://bucket/prefix, taking credentials from the environment and adding --endpoint-url for any store that is not Amazon S3. Re-run the same command to pick up changes: objects whose ETag and size are unchanged are not downloaded again. Report the LIST and GET counts the run prints before you put the command in a schedule, and remember the index itself is on the node's local disk, not in the bucket.
RUN THIS XERJ COMMAND xerj autoindex s3://acme-docs/handbook Index one prefix. Credentials come from AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the environment.
RUN THIS XERJ COMMAND export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... The environment is the only place XERJ reads keys from: no profile files, no instance metadata, no SSO.
RUN THIS XERJ COMMAND xerj autoindex s3://acme-docs/handbook --dry-run List and price the transfer without downloading anything.
RUN THIS XERJ COMMAND xerj autoindex r2://acme-docs/handbook --endpoint-url https://accountid.r2.cloudflarestorage.com Cloudflare R2, or any S3-compatible store. The account host has no default, so it is required.
RUN THIS XERJ COMMAND xerj autoindex s3://acme-docs/handbook --state-dir /data/xerj-state The mirrored object bytes live under the state dir. Point it at a disk with room.
RUN THIS XERJ COMMAND curl -s 'http://127.0.0.1:9200/ax-*/_search?q=onboarding&size=3' Query the node the run indexed into. The index is on local disk, not in the bucket.

The command

xerj autoindex s3://acme-docs/handbook

That is the whole change from indexing a folder. Everything after the listing is the pipeline you already have: format sniffing, the code and document extractors, the plan, the resume journal, the incremental reconcile.

For anything that is not Amazon S3, add the endpoint:

# MinIO, Ceph, SeaweedFS, localstack
xerj autoindex s3://mybucket/docs --endpoint-url http://127.0.0.1:9000

# Cloudflare R2 — the account host has no default, so it is required
xerj autoindex r2://mybucket/docs --endpoint-url https://accountid.r2.cloudflarestorage.com

--endpoint-url falls back to AWS_ENDPOINT_URL_S3, then AWS_ENDPOINT_URL, which is the precedence the AWS CLI uses.

Credentials come from the environment — AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN for temporary credentials — and from nowhere else. Profile files, instance metadata (IMDS) and SSO are not read; if your keys live in a profile, export them for the run. A URL of the form s3://key:secret@bucket/p is refused by name, because that shape puts a credential in every request line and in every log that records one.

One surprise removed on purpose: a non-empty prefix is treated as a folder. s3://b/docs lists docs/ and not also docs-old/, even though plain S3 prefix matching is a string prefix. The effective prefix is printed at the start of the run.

What makes the second run cheap

The ETag plus the size, recorded per object in <state-dir>/object-source.json.

On a re-run, an object whose ETag and size match what was recorded is not downloaded. That is the whole mechanism, and it is why pointing a nightly job at a bucket is affordable.

Three things XERJ refuses to assume about an ETag:

A second condition has to hold as well as the ETag matching: the mirrored file must still be there at the size the store reports. That is what makes a run that was killed half-way heal itself instead of indexing a truncated file.

What it costs in requests

Object stores bill per request. On Cloudflare R2's free tier that is 1,000,000 class-A operations a month (LIST and writes) and 10,000,000 class-B (GET, HEAD).

An autoindex run over a bucket performs no writes at all. No PutObject, no CopyObject, no bucket creation. Its only class-A cost is listing, at the protocol maximum of 1,000 keys per page:

Objects under the prefixLIST per runDailyHourlyEvery 5 min
1,0001307208,640
10,000103007,20086,400
100,0001003,00072,000864,000

Those are requests per month against a 1,000,000 free allowance, for that prefix alone. A daily or hourly re-index of a normal corpus is a rounding error. A minute-interval watcher over a six-figure bucket is not — and nothing in XERJ creates one: xerj autoindex is a single command with no polling loop and no watch mode. If you put it in cron, the interval is the cost.

You do not have to derive any of this. Every object-store run prints it:

autoindex: object store requests this run: 2 LIST (class A) + 1200 GET (class B), 0 MB transferred
autoindex: re-running this at the same size costs 2 class-A request(s) each time:
  60/month daily (0.0%), 1440/month hourly (0.1%), 17280/month every 5 minutes (1.7%)

Those lines survive --quiet, and the same counts land in the run document under object_source.

--dry-run lists and prices the transfer without downloading anything. Once the mirror is current there is nothing left to transfer, so a dry run then gives the full plan projection for the cost of one listing.

Where the bytes go, and what that costs you

Objects are streamed into a local mirror under the state directory, and the ordinary walk runs over that mirror.

The reason is not laziness. Several extractors are not sequential readers: a PDF is read from its trailer backwards, a DOCX is a ZIP whose central directory sits at the end of the file, SQLite seeks by page, and the content-identity contract reads each file twice before verifying it a third time. Over a network stream each of those becomes either another GET — more requests, more money — or a copy of the whole object in memory, which is the one thing that must not happen.

So the trade is explicit:

The index itself is on the XERJ node's local disk. Nothing writes to the bucket, and keeping the index in object storage is a different, unimplemented thing — see what XERJ does not do with S3 yet.

What is skipped, and why you should be glad

An object key is an arbitrary string, not a path. Keys that cannot become a safe, portable relative path are refused by name, and each rule is reported with a count:

RuleWhat it catches
hidden:dotfileany dot-prefixed component, so a bucket's .env, .git/config and .ssh/id_rsa stay out of a queryable index
object:unsafe-key.., ., an empty component (a//b), a leading slash, a backslash, a control character, a component over 255 bytes, a trailing dot or space, a Windows reserved name
object:case-collisiontwo keys differing only in case, which would be one file on macOS or Windows
object:folder-markerthe zero-byte docs/ entries every console creates
default:build-outputnode_modules/, vendor/, target/, dist/, build/, .venv/, __pycache__/

.gitignore and .xerjignore are the two exceptions to the dotfile rule: they are fetched so the bucket's own ignore rules still apply, and then not indexed.

A skipped key is also money not spent — the filter runs on the listing, so node_modules/ never costs a GET.

Deletions

An object that disappears from the bucket stops appearing in search results on the next run, through the reconcile machinery that already handles a deleted file. That machinery is the generated --no-graph journal. Index a bucket whose objects get deleted with xerj autoindex s3://bucket/prefix --no-graph. On the default graph-enabled journal the run refuses and writes nothing. It names the content groups that vanished and the ways to rebuild. That is the same limit a deleted file hits there, not an object-store restriction.

After listing, the mirror is walked and everything the store no longer lists is deleted from it. The walk then sees a smaller corpus and the incremental reconcile removes those documents. The mirror is walked rather than the recorded manifest diffed, on purpose. A lost manifest must not be able to leave a deleted object searchable forever.

One asymmetry worth knowing. An object that is listed and then 404s was deleted between the LIST and the GET. That race is normal on a live bucket, so the run treats it as deleted and carries on. Any other GET failure aborts the run before anything is indexed. Skipping a failed download would present that object to the reconcile step as deleted, and a transient 500 would then delete live documents.

An aborted run does not make you pay for its work twice. Objects whose bytes already landed are written to the manifest before the error is returned, and that manifest is checkpointed every ten seconds during a long transfer. The next run fetches only what is left. It does re-list, so the LIST cost is paid again in full.

How this page was checked

Most numbers above come from the MinIO suite in engine/crates/xerj-autoindex/src/objsource_minio_tests.rs, run against quay.io/minio/minio:latest on loopback. The last row is the shipped binary, end to end against a live XERJ node and the same MinIO. Every measurement on this page is MinIO on loopback, not Amazon S3 and not R2. The request arithmetic for R2 comes from Cloudflare's published pricing, not from a bill.

CaseMeasured
first run, four keys in the prefix1 LIST + 3 GET
unchanged re-run1 LIST + 0 GET, 0 bytes
one changed, one new, one deleted2 GET, 1 local removal
1,200 keys2 LIST pages
real multipart uploadETag ending -2, not re-downloaded
1 GiB objectRSS 22 MB to 26 MB; 281-318 ms over four loopback runs
end to end against a node, 6 keys incl. a 12 MiB multipartfirst run 1 LIST + 5 GET, searches hit; re-run 1 LIST + 0 GET; one change 1 LIST + 1 GET; one delete (--no-graph) removes only that object's documents

The request arithmetic in the table is multiplication, not measurement: LIST requests per run times runs per month.

FAQ

How do I search files that live in an S3 bucket?

Run xerj autoindex s3://bucket/prefix against a running node. It lists the prefix, streams each object into a local mirror under the state dir, and indexes the documents. Credentials come from the environment.

Does it download the whole bucket every time I run it?

No. Each object's ETag and size are recorded in the state dir, and a re-run downloads only what changed. Measured against MinIO, an unchanged prefix costs 1 LIST request and zero GETs.

Will this blow through my R2 or S3 free tier?

A run makes no writes, so its only class-A cost is listing: one request per 1,000 keys, plus any attempt the store throttled and made the client retry. The printed counts are billed wire attempts, not logical calls, so what a run reports is what the store bills. Daily or hourly over a normal corpus is a fraction of a percent, and every run prints what a schedule would cost.

Can I use MinIO, Ceph or Cloudflare R2?

Yes. Pass --endpoint-url, or set AWS_ENDPOINT_URL_S3. R2 needs the account host because there is no default for it. The bucket is then addressed in the path rather than as a subdomain.

Does the index live in the bucket?

No, and that matters for sizing. The objects are mirrored to local disk under the state dir and the index is written to the XERJ node's own disk. Keeping the index in object storage is not implemented.

What happens to search results when someone deletes an object?

Run with --no-graph. The next run removes the object from the local mirror and the generated journal's incremental reconcile deletes its documents. On the default graph-enabled journal a deletion is refused instead, exactly as it is for a folder, and the run tells you how to rebuild.

Is a multipart-uploaded object handled correctly?

Yes. A multipart ETag is a digest of digests with a -N suffix, so it is not an MD5 of the body. XERJ never treats it as one: it stores the string and asks only whether it changed.

My bucket has 40,000 objects. What does one run cost?

40 LIST requests, plus one GET per changed object, plus disk for the bytes. Hourly that is 28,800 class-A requests a month, under 3% of a 1,000,000 free allowance for that prefix.

Evidence

Related