--- title: "What is syntax-aware search?" canonical: "https://xerj.org/science/syntax-aware-search" updated: "2026-11-14" source: "content/science/syntax-aware-search.md" --- # What is syntax-aware search? **Short answer** (updated 2026-11-14): What is syntax-aware search? It indexes declared symbols from a parser, so you can look up a definition by kind, name and line, not only by matching raw text. XERJ uses tree-sitter during autoindex, and covers 36 languages, pinned by a unit test. A defs query returns declarations. A body query still finds raw text, including call sites. **Key number:** XERJ covers 36 languages via tree-sitter, pinned by a unit test. A file over 2 MiB is skipped, at most 5,000 symbols per file, and names over 200 bytes are dropped. (as of 2026-11-13; source: [/answers/syntax-aware-code-search-refactoring](/answers/syntax-aware-code-search-refactoring)) **TL;DR** Syntax-aware search indexes declared symbols from a parser, so you can look up a definition by kind, name and line, not only by matching raw text. XERJ uses tree-sitter for that step. ## Agent prompt ```text Act as a coding agent. Read https://xerj.org/llms.txt, install XERJ, start a local node, index one folder I point you to, and run one search so I can see a cited passage. ``` *Figure: Language count and caps follow engine/crates/xerj-autoindex/src/extract/code.rs, as published on the syntax-aware answer. Tree-sitter itself is documented at tree-sitter.github.io/tree-sitter/.* ## How it works A text query finds the characters you type, including comments and string literals. It cannot tell a declaration from a call site. A syntax-aware index walks a parse tree and writes declared symbols: kind, name and line. A definition query can then return those declarations. A separate text query still finds raw occurrences, including calls. The parser XERJ uses for this is tree-sitter, an incremental parser library. Official documentation: [tree-sitter.github.io/tree-sitter](https://tree-sitter.github.io/tree-sitter/). ## How XERJ uses it During `xerj autoindex`, tree-sitter writes each declared symbol as kind, name and line. A `defs` query, or `xerj def`, returns those declarations. A `body` query finds raw text. 36 languages are covered, pinned by a unit test. Caps, from `engine/crates/xerj-autoindex/src/extract/code.rs`: a file over 2 MiB is skipped, at most 5,000 symbols per file, and names over 200 bytes are dropped. There is no call graph. XERJ rewrites nothing. For a structural rewrite, use ast-grep or a language server. The how-to comparisons are [syntax-aware code search](/answers/syntax-aware-code-search-refactoring) and [XERJ vs ast-grep](/compare/xerj-vs-ast-grep). This is also the lookup step inside [reference coding](/science/reference-coding). ## History and research Tree-sitter is documented at the official site linked above. The language count and caps are XERJ's published figures, from the syntax-aware answers and the engine source. This page does not add a new language-count study. ## Guides Try it on real code: [syntax-aware code search for refactoring](/answers/syntax-aware-code-search-refactoring), [XERJ vs ast-grep](/compare/xerj-vs-ast-grep) and [semantic code search](/answers/semantic-code-search). ## Related terms [Reference coding](/science/reference-coding) and [Corpus Hub](/science/corpus-hub). ## FAQ ### What is syntax-aware search? It is search that knows declared symbols from a parser, so a definition query can return kind, name and line, not only a text match. ### How does XERJ build that index? During autoindex, tree-sitter writes each declared symbol as kind, name and line. A defs query, or xerj def, returns those declarations. 36 languages are covered. ### What if a file does not parse? If a grammar fails to parse a file, XERJ indexes that file as plain text, so a text query still works. Caps: a file over 2 MiB is skipped; at most 5,000 symbols per file. ### Does XERJ rewrite code? No. XERJ rewrites nothing. For a structural rewrite, use ast-grep or a language server. The comparison is the syntax-aware vs ast-grep answer. ## Evidence - XERJ covers 36 languages via tree-sitter. A file over 2 MiB is skipped, at most 5,000 symbols per file, and names over 200 bytes are dropped. — [/answers/syntax-aware-code-search-refactoring](/answers/syntax-aware-code-search-refactoring) ## Related - [Syntax-aware code search: XERJ vs ast-grep](/answers/syntax-aware-code-search-refactoring) - [ast-grep vs a code index for an agent?](/compare/xerj-vs-ast-grep) - [Reference coding for AI coding agents](/science/reference-coding) - [What is Corpus Hub?](/science/corpus-hub) - [BM25 vs vector search](/science/bm25) (same intent: Science) - [What is hybrid search?](/science/hybrid-search) (same intent: Science) - [What is semantic search?](/science/semantic-search) (same intent: Science)