The best local full-text search for JSONL and CSV files in 2026, with a published benchmark

TL;DR — For one JSONL or CSV file, an embedded index wins: SQLite FTS5 indexed 50,000 records in 0.29 s and answered in hundredths of a millisecond. ripgrep needs no index at all. XERJ returned the same counts but took 70.57 s to index, peaked at about 9.1 GiB, and was killed for running out of memory at 300,000 records.

Pick one in ten seconds

• Pick ripgrep if you search the file once or twice. Nothing to build, about 3 ms per search on 50,000 records.
• Pick SQLite FTS5 if you search the same file many times. It was the fastest to index and to query in our run.
• Pick DuckDB FTS if you also want SQL analytics over the same rows. Its index was the smallest, at 3.7 MB, but it must be rebuilt after the table changes.
• Pick tantivy-cli if you want a Lucene-style index from one JSON document per line without writing code.
• Pick qsv if your CSV questions are filters and regexes, not ranked search.
• Pick Meilisearch if you want a search server with typo tolerance for an app and can run it as a service.
• Pick XERJ if an agent searches a folder of mixed files through MCP, on a machine with plenty of memory.

Summary table

The benchmark

We ran four tools on one shared Linux host (8 CPUs, 16 GB RAM) on 2026-10-19. The corpus is deterministic synthetic log records written by gen_corpus.py, with planted terms only in the msg field. Each tool counted a rare term, a common term and a phrase. Latency is the median of 20 runs after 3 warm-ups.

All four returned identical counts. ripgrep's time includes a process spawn per query; SQLite and DuckDB are in-process calls; XERJ's includes an HTTP round trip over loopback. DuckDB has no phrase operator, so its phrase row requires all three terms; on this corpus the count is the same.

At 300,000 records (58.3 MB), ripgrep took 8.79–12.25 ms, SQLite FTS5 indexed in 1.84 s and DuckDB in 2.24 s, and all three returned 338 / 15,061 / 1,484. XERJ did not finish: the kernel killed the node at 9,972,316 kB of anonymous memory about 13 minutes in.

Scripts, raw JSON results, logs and the kernel OOM record are in benchmarks/best-of-jsonl-search-2026. Treat the absolute times as one machine's.

SQLite FTS5 and DuckDB: embedded and fast

SQLite FTS5 is a virtual table module, so the index lives in one database file next to your data. In our run it was the fastest to build and to query, at both sizes.

DuckDB reads JSONL directly with read_json_auto and builds a BM25 index with PRAGMA create_fts_index. Its index was the smallest. Its docs say the index does not update when the table changes, so rebuild it after new rows arrive.

tantivy-cli, qsv and Meilisearch: not benchmarked here

tantivy-cli indexes one JSON document per line into a tantivy index and serves queries from the command line; its README calls it suitable for smaller projects. qsv's search is a regex over CSV fields, and its index is for random row access, not full text. Meilisearch is a server that accepts NDJSON and CSV documents and adds typo tolerance.

We did not run these three, so the table shows "not run" rather than a guess.

XERJ: same answers, heavy indexing

XERJ returned the same counts as the other three, and once indexed it answered in about a third of a millisecond over HTTP. But indexing one 9.7 MB file took 70.57 s and peaked at 9,578,548 kB resident, and the 300,000-record run ran out of memory. On this workload that is a defect, and we publish it as measured.

XERJ still fits when an agent searches a folder of mixed files through one MCP server; the CLI walkthrough shows that path on a smaller folder.

How this page was checked

The benchmark ran on 2026-10-19 with the versions named in its table. Licences, install lines and capabilities of the tools not benchmarked were read from each project's README or docs the same day, and stars and latest releases came from the GitHub REST API. SQLite is not on GitHub; its latest release is from sqlite.org.

FAQ

What is the fastest way to search one JSONL file?

For a single search, ripgrep: no index, about 3 ms on 50,000 records in our run. For many searches over the same file, SQLite FTS5: a 0.29 s index build, then rare-term counts in 0.02 ms.

How was the benchmark run?

On 2026-10-19, on one shared Linux host with 8 CPUs and 16 GB of RAM, over deterministic synthetic log records with a rare term, a common term and a phrase. Latency is the median of 20 runs after 3 warm-ups. The scripts, corpus generator and raw results are in benchmarks/best-of-jsonl-search-2026 in the XERJ repository.

Why did XERJ lose this benchmark?

Indexing one 9.7 MB file took 70.57 s and peaked at about 9.1 GiB resident, and the 300,000-record run was killed by the kernel for running out of memory. That is a XERJ problem on this workload, published as measured. XERJ's query time once indexed was 0.31 to 0.35 ms over HTTP.

Can qsv do full-text search on CSV?

qsv's search command runs a regex over selected fields and prints matching rows. Its index command builds a row-offset index for fast random access, not a full-text index, so there is no ranking.

Do these tools rank results?

SQLite FTS5, DuckDB FTS, tantivy, Meilisearch and XERJ rank with BM25 or a similar score. ripgrep and qsv search print every matching line or row with no ranking.

When is XERJ still the right pick?

When an agent asks many questions across a folder of mixed files, such as JSONL next to CSV, logs and code, through one MCP server and one Elasticsearch-compatible API. For a single large JSONL file on a small machine, pick SQLite FTS5 or DuckDB.

Evidence

• On 50,000 synthetic JSONL records (9,724,646 bytes), ripgrep 14.1.1, SQLite FTS5 3.46.1, DuckDB FTS 1.5.6 and xerj v1.0.0-rc.97 returned identical counts of 51, 2,485 and 253 for a rare term, a common term and a phrase. — benchmarks/best-of-jsonl-search-2026/results-50k-2026-10-19.json
• SQLite FTS5 indexed the 50,000 records in 0.29 s into 13.8 MB, DuckDB FTS in 0.61 s into 3.7 MB, and XERJ in 70.57 s into a 112.3 MB data directory. — benchmarks/best-of-jsonl-search-2026/README.md
• At 300,000 records (58.3 MB) the kernel OOM-killed the XERJ node at 9,972,316 kB anonymous RSS about 13 minutes into indexing. — benchmarks/best-of-jsonl-search-2026/oom-300k-dmesg.txt
• DuckDB's full-text index does not update automatically when the table changes, and its BM25 scoring defaults to k=1.2 and b=0.75. — https://duckdb.org/docs/current/core_extensions/full_text_search
• tantivy-cli is the command-line interface for the tantivy engine, indexes one JSON document per line, and describes itself as suitable for smaller projects. — https://github.com/quickwit-oss/tantivy-cli
• qsv's search command runs a regex over selected CSV fields; its index command builds a row-offset index for random access, not a full-text index. — https://github.com/dathere/qsv
• Meilisearch installs with curl -L https://install.meilisearch.com | sh; the Community Edition is MIT and the Enterprise Edition is under a commercial licence or BUSL-1.1. — https://github.com/meilisearch/meilisearch
• Meilisearch accepts documents as JSON, NDJSON (application/x-ndjson) or CSV (text/csv). — https://www.meilisearch.com/docs/capabilities/indexing/getting_started
• In the JSONL benchmark of 2026-10-19 the XERJ node peaked at 9,578,548 kB (about 9.1 GiB) resident while indexing one 9.7 MB file of 50,000 records. — benchmarks/best-of-jsonl-search-2026/README.md
• At 300,000 records SQLite FTS5 indexed in 1.84 s into 74.4 MB and DuckDB FTS in 2.24 s into 15.5 MB; both, and ripgrep, returned 338, 15,061 and 1,484. — benchmarks/best-of-jsonl-search-2026/README.md

Related

• Full-text search over JSONL files from the CLI: index a folder and query it with BM25
• What's the easiest way to search JSON logs plus some old gzip text logs in the same folder?
• I have a directory of CSV exports. How do I query them without opening each one in Excel?
• What's a local alternative to Meilisearch that indexes a folder by itself?
• How do I index a CSV with many columns? · same intent: JSONL and data files
• I want to point something at a folder and then ask questions about what's in it. What should I use? · same intent: JSONL and data files
• How do I full-text search a SQLite database I just copied onto disk? · same intent: JSONL and data files