SCIENCE · AGENTS

What is context engineering?

AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-11-13 · UPDATED · 2026-11-14

SHORT ANSWER · UPDATED 2026-11-14

What is context engineering? It is choosing what an agent actually reads, so the context window is spent on the task. A common loop is to grep and open matching files. A tighter loop retrieves a cited passage and stops. On 8 tasks, the same model used 9,982 output tokens with XERJ against 26,477 when it grepped the source.

Key number: On 8 tasks, the same Claude Code model used 9,982 output tokens with XERJ against 26,477 when it grepped the source, 2.7 times fewer. (as of 2026-08-06 · source: /case-studies/reference-coding)

TL;DR Context engineering is choosing what an agent actually reads. The goal is a short, cited passage in the window, instead of a whole file, so the context window is spent on the task.

Try it: what the agent actually reads

A shorter context, with a citation

Many lines, many tokens

A common loop is to grep, open matching files, and read them. On psf/requests, one ripgrep listing was 25,150 bytes; opening the 17 matching files would have been 447,383 bytes.

Byte figures: benchmarks captured on psf/requests, published in the grep-token answer (2026-10-19). Token figures: the reference-coding case study. Do not scale one study into the other.

PASTE THIS TO YOUR AI AGENT — IT DOES THE REST

Act as a coding agent. Read https://xerj.org/llms.txt, install XERJ, start a local node, index one folder I point you to, and run one search so I can see a cited passage.

How it works

An agent has a limited context window. Every line it reads occupies that window. A common loop is to grep, open every matching file, and hope the useful passage is still in view.

A tighter loop retrieves a ranked passage with a file and line, and stops. The agent reads less, and the useful line is not buried in the middle of a long prompt.

Liu and colleagues (2023) showed that models often use the start and the end of a long context more reliably than the middle. The paper is Lost in the Middle: How Language Models Use Long Contexts. That result is one reason a shorter, cited passage is worth the extra retrieval step. It is not a XERJ measurement.

How XERJ uses it

XERJ is the retrieval step that makes the tighter loop possible. On psf/requests, one comparison recorded 1,303 bytes for a xerj search answer against a 25,150 byte ripgrep listing, measured once, on one repo. Opening the 17 matching files would have been 447,383 bytes. A byte is not a token.

On 8 coding tasks, the same Claude Code model used 9,982 output tokens with XERJ against 26,477 when it grepped the source, 2.7 times fewer, and solved 16 of 16 both ways. That study is the reference-coding case study. Do not scale the byte study into the token study.

The product pattern is reference coding.

History and research

The long-context result cited above is Liu et al., Lost in the Middle, 2023, later in TACL. The token and byte figures on this page are XERJ's published measurements, from the sources named above.

Guides

Guides: code search with fewer tokens than grep, index a monorepo and Claude Code memory.

FAQ