HOME·SCIENCE·WHAT IS SYNTAX-AWARE SEARCH?
SCIENCE · CODE AND CORPORA

What is syntax-aware search?

AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-11-13 · UPDATED · 2026-11-14
SHORT ANSWER · UPDATED 2026-11-14

What is syntax-aware search? It indexes declared symbols from a parser, so you can look up a definition by kind, name and line, not only by matching raw text. XERJ uses tree-sitter during autoindex, and covers 36 languages, pinned by a unit test. A defs query returns declarations. A body query still finds raw text, including call sites.

Key number: XERJ covers 36 languages via tree-sitter, pinned by a unit test. A file over 2 MiB is skipped, at most 5,000 symbols per file, and names over 200 bytes are dropped. (as of 2026-11-13 · source: /answers/syntax-aware-code-search-refactoring)

TL;DR Syntax-aware search indexes declared symbols from a parser, so you can look up a definition by kind, name and line, not only by matching raw text. XERJ uses tree-sitter for that step.

Try it: two queries on one index

Declarations vs text

Where is this declared?

During autoindex, tree-sitter writes each declared symbol as kind, name and line. A defs query, or xerj def, returns those declarations. 36 languages are covered, pinned by a unit test.

Language count and caps follow engine/crates/xerj-autoindex/src/extract/code.rs, as published on the syntax-aware answer. Tree-sitter itself is documented at tree-sitter.github.io/tree-sitter/.
PASTE THIS TO YOUR AI AGENT — IT DOES THE REST Act as a coding agent. Read https://xerj.org/llms.txt, install XERJ, start a local node, index one folder I point you to, and run one search so I can see a cited passage.

How it works

A text query finds the characters you type, including comments and string literals. It cannot tell a declaration from a call site.

A syntax-aware index walks a parse tree and writes declared symbols: kind, name and line. A definition query can then return those declarations. A separate text query still finds raw occurrences, including calls.

The parser XERJ uses for this is tree-sitter, an incremental parser library. Official documentation: tree-sitter.github.io/tree-sitter.

How XERJ uses it

During xerj autoindex, tree-sitter writes each declared symbol as kind, name and line. A defs query, or xerj def, returns those declarations. A body query finds raw text. 36 languages are covered, pinned by a unit test. Caps, from engine/crates/xerj-autoindex/src/extract/code.rs: a file over 2 MiB is skipped, at most 5,000 symbols per file, and names over 200 bytes are dropped. There is no call graph.

XERJ rewrites nothing. For a structural rewrite, use ast-grep or a language server. The how-to comparisons are syntax-aware code search and XERJ vs ast-grep.

This is also the lookup step inside reference coding.

History and research

Tree-sitter is documented at the official site linked above. The language count and caps are XERJ's published figures, from the syntax-aware answers and the engine source. This page does not add a new language-count study.

Guides

Try it on real code: syntax-aware code search for refactoring, XERJ vs ast-grep and semantic code search.

Related terms

Reference coding and Corpus Hub.

FAQ

What is syntax-aware search?

It is search that knows declared symbols from a parser, so a definition query can return kind, name and line, not only a text match.

How does XERJ build that index?

During autoindex, tree-sitter writes each declared symbol as kind, name and line. A defs query, or xerj def, returns those declarations. 36 languages are covered.

What if a file does not parse?

If a grammar fails to parse a file, XERJ indexes that file as plain text, so a text query still works. Caps: a file over 2 MiB is skipped; at most 5,000 symbols per file.

Does XERJ rewrite code?

No. XERJ rewrites nothing. For a structural rewrite, use ast-grep or a language server. The comparison is the syntax-aware vs ast-grep answer.

Evidence

Related