--- title: "How do I search Kubernetes pod logs?" canonical: "https://xerj.org/answers/search-kubernetes-pod-logs-on-disk" updated: "2026-08-21" source: "content/answers/search-kubernetes-pod-logs-on-disk.md" --- # How do I search Kubernetes pod logs? **TL;DR** — Collect the pod logs to local disk first, then run `xerj autoindex` on the folder. In a captured run, 1 `match_phrase` request returned 18 hits spread 6, 6 and 6 across 3 pod log files. XERJ is not the collector, and the CRI format was read as prose text. ## Agent prompt ```text Act as a coding agent. Read https://xerj.org/llms.txt, copy or stream the pod logs you care about into a local folder that keeps the var/log/pods/__//0.log layout, start a local XERJ node, run `xerj autoindex ./pod-logs --url http://127.0.0.1:9200 --prefix pl --progress plain`, POST a match_phrase on body with size 100, and report the per-file hit counts read from the returned hits rather than from an aggregation. ``` ## Commands ### Command 1 Note: Index a folder of already-collected pod log files from local disk. ```sh xerj autoindex ./pod-logs --url http://127.0.0.1:9200 --prefix pl --progress plain ``` ### Command 2 Note: Run 1 phrase query across every pod, and read the spread from the hits. ```sh curl -s -XPOST http://127.0.0.1:9200/pl-*/_search -H 'content-type: application/json' -d '{"query":{"match_phrase":{"body":"tapir connection reset"}},"size":100,"_source":["ax_path","ax_locator"],"track_total_hits":true}' ``` ### Command 3 Note: Read the family XERJ gave each log file, and the document count per file. ```sh curl -s -XPOST http://127.0.0.1:9200/autoindex-catalog/_search -H 'content-type: application/json' -d '{"query":{"bool":{"filter":[{"term":{"doc_kind":"file"}}],"must":[{"wildcard":{"path":"*var/log/pods*"}}]}},"size":10,"_source":["path","format","status","records"],"sort":[{"path":"asc"}]}' ``` ## Get the logs onto disk XERJ indexes files that already exist on a filesystem. Use `kubectl logs`, a node-level DaemonSet or your existing agent to write the pod logs into a folder, then index that folder. The captured fixture kept the kubelet layout, `var/log/pods/__//0.log`, and the CRI line format, ` `. Each of the 3 files held 240 lines. ```sh xerj autoindex ./pod-logs --url http://127.0.0.1:9200 --prefix pl --progress plain ``` The run read 3 files into 1 dataset and 57 documents live in `pl-docs`, with 0 junk files. Each log file produced 18 documents. ## One query covers every pod A single phrase query across `/pl-*/_search` returned 18 hits. The hits named all 3 files, 6 hits each, and each hit carried the full pod path. | pod log file | hits | | --- | --- | | `var/log/pods/default_checkout-7d9f8b6c4-2xk7l_.../checkout/0.log` | 6 | | `var/log/pods/default_billing-5c8d7a9b3-qq4mz_.../billing/0.log` | 6 | | `var/log/pods/payments_ledger-6b4c9d2e8-h7t3v_.../ledger/0.log` | 6 | ```sh curl -s -XPOST 'http://127.0.0.1:9200/pl-*/_search' \ -H 'content-type: application/json' \ -d '{"query":{"match_phrase":{"body":"tapir connection reset"}},"size":100,"_source":["ax_path","ax_locator"],"track_total_hits":true}' ``` Read the per-pod spread from the returned hits. Set `size` large enough to hold every hit. The terms aggregation on the same request also reported 6 per file, but no general rule follows from 1 agreement. ## The CRI format is read as prose, not as logs All 3 files were detected as `txt-prose` in the `autoindex-catalog`. The `logs` family did not fire on this format, so no `ts`, `level` or `message` field was produced. Plan for that. The line text is searchable through `body`, so a full-text query for `level=error` finds the lines that hold that string. No typed level filter and no date range query exist on this index. If you want typed fields, convert the lines to JSON or JSONL before you index them. Keep every JSONL line under 4,096 characters. A captured reproduction indexed 30 documents at 4,095 characters per line and 0 at 4,096, with the family flipping from `jsonl` to `json`. ## What XERJ is not doing here XERJ is not a log collector, not a log shipper and not a cluster component. The node fetched nothing over the network and observed 0 distinct non-loopback peers across 185 samples over its whole life. No container runtime and no cluster existed on the capture host. The 3 log files use the real kubelet path layout and the real CRI line format, but the fixture generator wrote them. The claim under test was reading such files from disk after you gather them, and that claim holds. ## What this capture does not tell you about scale The captured run indexed 3 files and 720 log lines on 1 host. That size demonstrates the cross-pod query and the family assignment, not log-scale throughput. XERJ runs single-node here, with no replication and no failover. Full-text ranking on `body` uses BM25. The default embedder in XERJ is lexical feature hashing, so a query and a paraphrase that share no words do not match. Neural embeddings are opt-in through `--embed-mode neural`. Every number above comes from RUN-B, captured on 2026-08-21. The binary was a `ci-test` profile build, so no wall-clock figure from this run is published as a performance number. ## FAQ ### Can 1 query search logs from several pods at once? Yes. In the captured run 1 `match_phrase` request returned 18 hits spread 6, 6 and 6 across 3 pod log files, with the file path on every hit. ### Does XERJ collect logs from a cluster? No. XERJ is not a log collector and not a log shipper. Copy or stream the logs to disk first with `kubectl logs`, a DaemonSet or your existing agent. ### Does XERJ parse the timestamp and level from a pod log? Not for the CRI format. All 3 files were detected as `txt-prose`, so no `ts`, `level` or `message` field was produced. The line text is searchable as `body`. ### How do I count matches per pod? Read the counts from the returned hits with a large `size`, then group on `ax_path` in your client. The captured run did that and got 6 per file. ### Did the aggregation agree with the hits in this run? Yes. The terms aggregation on the same request reported 6 documents per file, the same as the hits. Compare both before you publish a count. ### What if I convert the logs to JSONL first? Keep every line under 4,096 characters. A captured reproduction indexed 30 documents at 4,095 characters per line and 0 documents at 4,096. ### Where do these results come from? All results come from run RUN-B, captured on 2026-08-21 on a 16-core AMD EPYC 9645 host. ## Related - [What's the easiest way to search JSON logs plus some old gzip text logs in the same folder?](/answers/search-json-and-jsonl-logs) - [How do I search gzipped logs without decompressing the whole file — including when JSONL sits next to them?](/answers/search-gzip-logs-without-zgrep) - [What can I use for low-volume log search?](/answers/cheap-low-volume-log-search)