How should an agent figure out what's in a messy data folder before searching?

TL;DR — xerj autoindex map prints the catalog that the last run inferred. In our capture the map described 2 datasets from a 6-file folder. Each dataset carried every field with its type, its cardinality, its null share and a ready-to-send query.

Index the folder, then print the map

xerj autoindex writes a catalog while it indexes, and xerj autoindex map prints that catalog. The Markdown form is for a person and the --json form is for an agent.

xerj autoindex map --url http://127.0.0.1:9410 --disable-feedback

xerj autoindex map --json --url http://127.0.0.1:9410 --disable-feedback

What the map lists for a folder

The map opens with the run header, then lists one row per dataset, then one field table per dataset. Our 6-file folder produced 2 datasets in 0.2 seconds.

The document counts in that table come from a capture taken before XERJ began writing one document per code declaration. A source file now contributes one document for every declaration it holds on top of its whole-file document, so the dataset carrying the code file produces more documents than the row shows. The CSV row is unaffected. Read the count from your own map.

Each field row carries the inferred type, an estimated cardinality, a null share and 3 example values. The table below is the mx-data table from the capture.

Time ranges appear only with a date field

The map prints a time field and a time range when schema inference typed a field as a date. Our 6-file folder had none, so both columns printed as a dash.

Other datasets in the same run did have one, and the map printed the real bounds.

There is no date-typed field for a file modification time on a text document. A text dataset such as mt-docs mapped 9 fields, and 0 of them was a date. A content query therefore cannot order by file date.

No correlation section appeared

The map printed no correlation between datasets for this folder. What it printed instead was a graph line naming 5 live edges in .xerj-memory-mixed6-edges. A notes block explained why body became the elected text field.

State that limit before you promise a reader cross-dataset correlations. The capture is the source, and the capture has none.

Every file and every refusal

The catalog holds 1 document per file with its detected format, its status and the reason for a refusal. Query the catalog directly for the full picture.

curl -s -XPOST 'http://127.0.0.1:9410/autoindex-catalog/_search' -H 'content-type: application/json' -d '{"query":{"term":{"doc_kind":"file"}},"size":500,"_source":["path","format","status","reason","records"],"sort":[{"path":"asc"}],"track_total_hits":true}'

The whole capture cataloged 43 files: 39 indexed and 4 refused. The 4 refusals were an expanded DOCX, an image-only PDF, and 2 SQLite sidecar files. Each refusal carried its own reason string.

One catalog entry per dataset slug

The catalog stores 1 dataset document per slug, with the document id ds:<slug>. The capture indexed 18 folders into 1 node, and the catalog ended with 11 dataset documents against 23 non-hidden indices. A later folder with the slug docs had replaced the earlier one.

Give each folder its own --prefix, and print the map straight after each run if you want a per-folder view. Read _cat/indices when you want the full index list.

The edges index per folder

autoindex creates 1 .xerj-memory-<brain>-edges index per indexed folder by default, named after the folder. The capture finished with 18 of them. A first read of _cat/indices is therefore longer than a reader expects.

That index also holds a brain metadata document with no src field. Add {"exists": {"field": "src"}} to any query over it, or the first hit is meaningless.

Scope of this capture

The capture ran on 1 single-node process on 2026-08-21, over deterministic fixtures covering 43 cataloged files. This run measured no whole disk. Read the map as a description of the output and not as a capacity claim.

FAQ

I dropped a messy data folder on an agent. How should it figure out what's inside before searching?

Run xerj autoindex on the folder, then xerj autoindex map. The map names every dataset it inferred, the fields and their types, and the files it refused, before any query is sent.

Is there a tool that can look at a folder and tell me what's actually in it?

xerj autoindex map prints the catalog XERJ inferred: one row per dataset with its index name, formats, document count, time field and time range, then a field table per dataset.

How do I inventory a folder of mixed files without opening each one?

Query autoindex-catalog for doc_kind file. Our run listed 43 files, 39 indexed and 4 refused, and each refusal carried its reason string.

Can an agent read the map without parsing Markdown?

Yes. Run xerj autoindex map --json and the same catalog arrives as documents, with a fields_json string and ready-to-send queries per dataset.

Why does the map list fewer datasets than _cat/indices?

Because the catalog stores 1 dataset document per slug, with the id ds:<slug>. A later folder using the slug docs replaces the earlier entry.

Can I sort a search by file modification date?

No. A text dataset maps 9 fields and none of them is a date, so no indexed file-date field exists to sort on.

What are the .xerj-memory indices in _cat/indices?

autoindex creates 1 .xerj-memory-<brain>-edges index per indexed folder by default. Our run finished with 18 of them.

Related

• I have a folder of PDFs, Word docs, and markdown. How do I search all of them at once?
• I have a directory of CSV exports. How do I query them without opening each one in Excel?
• How do I read autoindex progress?
• Zero-config autoindex