HOME·ANSWERS·I EXPORTED MY GMAIL WITH GOOGLE TAKEOUT. HOW DO I SEARCH THAT MBOX FILE LOCALLY?
ANSWERS · FOLDER INDEXING

I exported my Gmail with Google Takeout. How do I search that mbox file locally?

AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-09-18 · UPDATED · 2026-09-20

TL;DR — Unzip the Takeout download, run xerj autoindex on the folder with its own --prefix, then search the mailbox over HTTP. The mbox is recognised by its content and streamed one message at a time. Attached PDFs become per-page documents linked to their message. Senders are filterable by bare address, and Gmail thread ids and reply headers become filters and graph edges. With the default embedder, which is lexical (feature hashing, not neural), no message text leaves your machine. The node's memory is the limit today: at a 16 GiB laptop's memory cap, a 300 MB mailbox already drove the node past that laptop's RAM in our runs.

PASTE THIS TO YOUR AI AGENT — IT DOES THE REST Act as a coding agent. Read https://xerj.org/llms.txt. The user has a Google Takeout download, which is private mail. Unzip it first (autoindex does not open archives), start a node with `xerj --insecure --data-dir ~/xerj-data` on the user's own machine, run `xerj autoindex <unzipped folder> --url http://127.0.0.1:9200 --prefix mail --progress plain`, and relay the xerj-bar line while it runs. Before indexing, tell the user the node's memory limit (#948): at a 16 GiB laptop's cap, a 300 MB mailbox drove the node to 18-20 GiB. Answer questions with term filters on email_from_address / email_thread_id (hex) / attachment_name and match or match_phrase on body, which also holds the subject. Cite ax_locator and email_subject for every hit. If you file a field report, pass --pointed-at "a private mailbox" so the folder path is not published.
RUN THIS XERJ COMMAND bash -c 'unzip takeout-20260101T000000Z-001.zip -d ~/mail-export' Extract the download first. An unextracted .zip is named in the run's output with the exact command to run.
RUN THIS XERJ COMMAND xerj --insecure --data-dir ~/xerj-data Start a local node with no authentication on 127.0.0.1. Keep the data dir outside the folder you index. On a shared machine, drop --insecure and use the admin key.
RUN THIS XERJ COMMAND xerj autoindex ~/mail-export --url http://127.0.0.1:9200 --prefix mail --progress plain Index the export. The mbox is detected by content and streamed, and the progress bar moves inside the file.
RUN THIS XERJ COMMAND curl -s -XPOST http://127.0.0.1:9200/mail-*/_search -H 'content-type: application/json' -d '{"query":{"bool":{"filter":[{"term":{"email_from_address":"alice@example.org"}}],"must":[{"match_phrase":{"body":"signed lease"}}]}},"_source":["email_subject","email_date","ax_locator"]}' Ask the mailbox a question, scoped to one sender by bare address.
RUN THIS XERJ COMMAND curl -s -XPOST http://127.0.0.1:9200/mail-*/_search -H 'content-type: application/json' -d '{"query":{"bool":{"filter":[{"term":{"attachment_content_type":"application/pdf"}}],"must":[{"match_phrase":{"body":"INV-2024-0117"}}]}},"_source":["attachment_name","email_subject","ax_locator"]}' Find the page of an attached PDF that mentions something, and the mail it came with.

Unzip first

Google Takeout hands you one or more .zip (or .tgz) files. autoindex does not open archives, so extract them before you index.

An unextracted archive is recorded as junk, and the run prints it under its summary with the command to run. The line starts not indexed — archives are never opened and names the file. The same list is in the --json summary as unextracted_archives. The run goes on with whatever else is in the folder.

unzip takeout-20260101T000000Z-001.zip -d ~/mail-export

The result is a Takeout/ folder. It holds Mail/All mail Including Spam and Trash.mbox, an archive_browser.html index page, and the other products you exported.

Index the folder

xerj --insecure --data-dir ~/xerj-data
xerj autoindex ~/mail-export --url http://127.0.0.1:9200 --prefix mail --progress plain

--insecure starts the node with no authentication on 127.0.0.1. On your own laptop that is fine. On a machine other people use, anyone there can read the mailbox while the node runs. In that case start the node without the flag, and pass the key it writes to ~/xerj-data/admin.key as XERJ_API_KEY="$(cat ~/xerj-data/admin.key)".

autoindex recognises the mailbox by its content: a From <sender> <date> separator line followed by mail headers. It does not look at the .mbox extension. A renamed or extension-less mailbox is still a mailbox, and a text file called notes.mbox is not.

The file is streamed. Messages are parsed on a pool of --workers threads, and the output is forwarded in message order. The bytes of messages in flight are capped at 256 MB for the whole process. The progress bar advances inside the file, so a one-file corpus does not sit at zero for the whole run.

A folder that holds archive_browser.html is treated as a Takeout root. That page is skipped. A Keep note's .html twin is skipped when the .json beside it is the note. Both rules are named in the ignore accounting the run prints, and both are off under --no-default-ignores. Everything else in the export is indexed by ordinary sniffing.

What comes back

Every message becomes a document with the decoded plain-text body and these fields: email_from, email_to, email_cc, email_subject, email_date, email_message_id, email_in_reply_to, email_references, email_thread_id and email_labels.

Two details decide whether a query works:

Gmail's X-GM-THRID is stored in email_thread_id as lower-case hex, the form the Gmail web interface uses. The export writes it in decimal, so a filter on the decimal you see in the mbox finds nothing. Convert it first with printf '%x\n' 2301773278856733157, which prints 1ff18a9a0fbde1e5. email_labels holds Gmail's X-Gmail-Labels.

Every attachment becomes its own document: one per page for a PDF, sectioned text for a text attachment, and a name/type/size card for anything else. An attachment document carries attachment_name, attachment_content_type and attachment_bytes, plus the parent's message id, subject and sender. A PDF page is titled with the attachment's name.

curl -s -XPOST http://127.0.0.1:9200/mail-*/_search -H 'content-type: application/json' -d '{
  "query": {"bool": {"filter": [{"term": {"attachment_content_type": "application/pdf"}}],
                     "must": [{"match_phrase": {"body": "INV-2024-0117"}}]}},
  "_source": ["attachment_name", "email_subject", "email_from", "ax_locator"]}'

ax_locator is positional and stable. m<offset>-msg-s0 is a message, and m<offset>-att2-p7-s0 is page 7 of its third attachment. A long body is cut into sections that overlap by a paragraph, so a word in that shared paragraph can come back from two sections of the same message. Collapse on email_message_id when you want messages, not sections. Re-running the command overwrites by the same ids instead of duplicating.

Threads become a graph

The email-thread@1 detector writes two edge types. A replies_to edge comes from the In-Reply-To header. When that header is missing or names a message outside the export, the nearest ancestor in References is used. An attachment_of edge goes from every attachment document to the message that carried it.

Each edge quotes the header it came from. A parent that is not in the export is counted as unresolved and never invented. Subject-line threading is deliberately not attempted.

The edges are a function of the whole corpus, not of what one run re-read. When only one mailbox changed, the run loads the other mailboxes' messages back from the index before it resolves replies. A reply in Inbox to a message in an untouched Sent therefore survives. The brain is bi-temporal: a superseded edge is kept with invalid_at set. To see live edges only, add "must_not": [{"exists": {"field": "invalid_at"}}] to an edge query.

Privacy

Limits, stated plainly

The mailbox splitter and the message extractor were tested against a generated mailbox. It contains what real exports contain: quoted and unquoted From lines in bodies, 8-bit bodies with and without a declared charset, encoded-word subjects, and a truncated multipart. It also plants a 300 KB line, a duplicated Message-ID, and a final message with no trailing newline.

The Takeout layout rules have only seen that synthetic tree. They are not verified on a real Takeout export. Outlook PST/OST and Maildir have no extractor: convert them to mbox with a tool of your choice first. We have not tested any converter, so we do not recommend one. Undeclared Cyrillic or CJK legacy encodings are not detected.

Server memory is the limit today. On the 1 GB synthetic mailbox, xerj autoindex itself peaked at 296 MiB and finished in 279 s. All 2,530 planted needles were found in the right message. The node needed 66.9 GiB of peak memory to get there, with its process cap lifted.

Under the default cap on the same machine (16 GiB), the server sat at its memory watermark from 87 % of the mailbox on. The run aborted after ten minutes of waiting, with 82,422 of 106,581 documents indexed.

A 16 GiB laptop's default cap is 8 GiB. A 300 MB synthetic mailbox run at that cap on the same large machine completed twice, and every needle was found. But the server's peak memory was 18.2 GiB and 19.9 GiB, which is more than that laptop has.

The breaker stops accepting new work at the cap, but work already accepted keeps growing. We have not run it on a 16 GiB machine. This is the engine's memory while it indexes, filed as #948.

Until it is fixed, do not count on a 16 GiB laptop for more than a small mailbox. We have not measured where that limit is, only that 300 MB is past it on our machine.

Searching after a restart is uneven. We reopened the same 1 GB index on a fresh node under the default cap. Most first-time questions took under 50 ms, and others took several seconds. In two measurements, 16 and 17 of 40 queries took over a second, and the slowest took 16.6 s.

With the cap lifted, 4 of 40 did. Every answer was correct. The machine was shared during those runs, so treat the seconds as upper bounds. This is reported on #948 as well.

Wall time, throughput, memory and index size are in the repository's benchmarks/mbox-ingest/README.md, with the machine and the exact commands.

FAQ

How do I search a Google Takeout mbox file locally?

Unzip the download, start a local XERJ node, and run xerj autoindex <folder> --prefix mail. The mbox is detected by content and streamed. Then query mail-* over HTTP: filter on email_from_address, email_thread_id or attachment_name, and use match or match_phrase on body.

Why does a term filter on email_from find nothing?

email_from holds the header as a person reads it, such as Alice Anders <alice@example.org>, and it is a keyword, so only the whole string matches. Filter on email_from_address (or email_to_address, email_cc_address) with the bare, lower-case address.

The mbox is 8 GB. Does it get loaded into memory?

Not by autoindex. The splitter streams the file, and the client peaked at 296 MiB on a 1 GB mailbox. The node is the limit today. On that 1 GB mailbox the server needed 66.9 GiB with its cap lifted and did not finish under its default 16 GiB cap. At a 16 GiB laptop's 8 GiB cap, a 300 MB mailbox completed, but the server peaked at 18–20 GiB. That is filed as #948. Do not expect a multi-GB export to finish on a laptop until it is fixed.

Can it search inside the PDFs people attached?

Yes. A PDF attachment goes through the PDF extractor and becomes one record per page, titled with the attachment's name and carrying the parent's email_message_id, email_subject and email_from_address. An attachment_of edge points back to the message.

Does it keep the Gmail threads?

Yes. Gmail's X-GM-THRID is stored as email_thread_id in lower-case hex, the form the Gmail web interface uses. The export writes it in decimal, so convert with printf '%x\n' <decimal> before you filter. In-Reply-To and References become replies_to edges in the brain, so a whole thread is one term filter and a reply chain is a graph walk.

Does my mail leave my machine?

Not with the default configuration. The default embedder is lexical (feature hashing, not neural) and runs inside the node, and autoindex talks only to the node URL you give it. --embed-mode proxy would send body text to the endpoint you configure. A field report filed with xerj feedback is public and auto-fills the indexed folder's path, so for private mail pass --pointed-at "a private mailbox". --insecure means no authentication for anyone on the same machine.

Do I have to unzip the Takeout download?

Yes. autoindex does not open archives. An unextracted .zip or .tgz is recorded as junk, the run prints it under its summary with the exact command to run, and the run continues.

Has this been tested on a real Takeout export?

No. The mbox splitter and the message extractor are tested against a generated mailbox that reproduces the hard cases real exports contain. The Takeout folder layout rules have only seen that synthetic tree. Treat them as unverified on a real export until someone runs one.

Evidence

Related