Open Source Project #240: XERJ — Let Your Coding Agent Query an Index Instead of Reading Files, 40x Fewer Tokens Than grep

Introduction

"An agent that greps pulls whole files into its context. An agent that queries XERJ gets the exact passage instead."

This is article #240 in the "One Open Source Project a Day" series. Today's project is XERJ — a locally-run, Elasticsearch-API-compatible AI search engine built by the xerj-org team, 7,015 Stars, Apache-2.0 license, and a Trendshift trending repository.

AI coding agents run into a persistent problem: understanding an unfamiliar codebase usually means grepping for a hit, then reading the entire file that contains it — meaning you burn a token budget on an entire file even when you only need one function. WordPress core's PHP alone runs roughly 5.2 million tokens, the equivalent of 26 full context windows — an agent simply cannot "just read" its way through that. XERJ's answer is direct: autoindex any folder with one command — code, docs, logs, and PDFs all work — and the agent gets back the exact passage it needs on query, instead of a whole file.

What You'll Learn

• XERJ's flagship use case, "Reference Coding": having the agent retrieve how similar open-source projects already solved a problem before it writes its own code
• How autoindex works — zero schema, zero pipeline configuration, one command across dozens of formats including code, docs, logs, PDFs, and Unity projects
• Why it chose to be Elasticsearch-API-compatible instead of inventing a new protocol
• Quick start: from a one-line install to wiring it into MCP
• A distinctive "membership card" mechanism — one field report in exchange for a free Apache-2.0 binary

Prerequisites

• Basic hands-on experience with an AI coding agent (Claude Code, Cursor, etc.)
• Familiarity with Elasticsearch basics (indices, query DSL, aggregations) helps but isn't required
• A rough understanding of RAG (retrieval-augmented generation) helps frame the use cases

Project Background

Overview

XERJ bills itself as a "community-trusted local AI search" engine: its core capability is automatically indexing any folder so a coding agent stops reading files one by one and queries for the exact code it needs instead. Its biggest differentiator is being fully compatible with the Elasticsearch REST API — existing Elasticsearch client libraries, Kibana, and the surrounding tooling connect to XERJ unchanged — while the implementation underneath is a single static Rust binary, no JVM, no dependencies.

Author / Team

• Developed by: xerj-org, a dedicated organization account
• Primary language: Rust
• License: Apache-2.0
• Created: 2026-06-30
• Status: Actively developed, a Trendshift trending repository

Project Stats

• ⭐ GitHub Stars: 7,015+
• 🍴 Forks: 356+
• 📄 License: Apache-2.0
• 📅 Created: 2026-06-30
• 👥 Core contributors: the xerj-org organization account itself leads with 2,742 commits, followed by Vinz2168 (130), buger (109), thomas-villani (38), and other active contributors
• 📬 Open Issues: 82
• 🏷️ Topics: ai-search, ai-search-engine, bm25-search, elasticsearch, graph-search, hybrid-search, semantic-search, vector-database, vector-search, and more
• ✅ Elasticsearch conformance suite runs on every commit — currently 1,366 of 1,369 cases pass, with the remaining gaps listed openly in the repo rather than hidden

Quick Start

One-Line Install

curl -fsSL https://xerj.org/get | sh

Windows PowerShell:

irm https://xerj.org/get.ps1 | iex

One static binary, no JVM, no dependencies — prebuilt for Linux, macOS, and Windows on x86-64 and arm64. You can also build from source.

Start and Index Your First Project

xerj --insecure --data-dir ./data &              # local dev mode: no TLS, no auth
xerj init                                        # wires XERJ into Claude Code/Cursor (MCP + skill), one command
xerj autoindex ~/my-project                      # index any folder
xerj search "how do we rotate the WAL segment"   # ranked passages, with file:line
xerj def "euler_to_rotationmatrix"               # go-to-definition: file:line + signature
xerj gain                                        # what it did for you — counted, not estimated

A note worth flagging about --insecure: it turns off TLS and API-key auth, so every caller that can reach the port becomes a superuser — able to read and delete every index, including every /_memory namespace. This is a deliberate, scoped trust boundary: it's meant for exactly one user on a dev machine — a fresh node binds only to 127.0.0.1, and a cleartext bind on any other interface refuses to start unless explicitly allowed. The full model is documented in docs/SECURITY_MODEL.md.

Have Your Agent Install It For You

If your agent can run shell commands, you don't even need to run the above by hand — just give it this prompt:

Install XERJ (docs: https://xerj.org/llms.txt), index this project's sources, and set up
reference coding: clone and index the open-source repos closest to what we're building,
and search how they solved a problem before writing code.

This relies on llms.txt — a structured install/usage document written specifically for agents to parse, a convention more and more agent-facing tools are adopting.

Core Deep-Dive

Reference Coding: Have the Agent "Copy" Before It Writes, Instead of Working From Memory

XERJ's flagship, best-measured use case is called "Reference Coding": give an agent a task, and it automatically downloads the open-source repos closest to that problem, indexes them, and queries "how did these projects solve this before" before writing a line of its own code — reusing that prior art instead of generating purely from the model's memory.

The project ran a controlled study (8 tasks, 4 languages, 16 runs per arm, real claude -p token counts):

At the same 16/16 solve rate, XERJ used 2.7x fewer output tokens than the grep-driven arm, and 26x fewer than memory alone — with a gap as wide as 278x on a single Java task. The project is also upfront about the limits of this win: the benefit is gated by memorization — it wins hardest on private, internal, niche, or post-cutoff code, and shrinks toward neutral-to-harmful on popular public libraries the model already knows well. Stating that boundary explicitly, instead of touting the headline number unconditionally, is not something every performance-driven project bothers to do.

Autoindex: Zero Schema, Dozens of File Formats Handled Out of the Box

XERJ's autoindex command sniffs every file, works out what it is, and creates one independent index per dataset it discovers — no schema to write, no pipeline to configure:

phase A: 593 datasets inferred, 1955 junk/skipped files
phase B: indexing 25329 files with 8 workers
done in 158.1s, 593 datasets, 83103 records live, 790 junk records

Source files are parsed with tree-sitter and arrive in the index with their symbols and line numbers intact, rather than as flat text — meaning what the agent gets back on query is a function, not a line. The list of supported formats is long: CSV, JSON, JSONL, XML, YAML, SQLite, PDF, DOCX, PowerPoint (one record per slide), Excel (one dataset per sheet), EPUB (one record per chapter), Jupyter notebooks (split by heading section), man pages, HTML, mail (including Google Takeout's mbox exports), and more. Worth calling out specifically: Unity projects get first-class treatment — text-serialized scenes, prefabs, and assets become one record per GameObject/Component, .meta files become a GUID-to-path table, and MonoBehaviour records carry script_class/script_path, turning "which scenes use this script?" into a single query.

Re-indexing is incremental: a file whose content is byte-identical to the last indexed generation skips parsing entirely and carries its committed result forward — the common edit-and-rerun / CI case, where typically over 95% of files are unchanged, does work proportional to what actually changed rather than the whole repo. The total re-index time ends up floored by content hashing, not parsing.

Why Elasticsearch Compatibility Instead of a Custom Protocol

XERJ didn't invent a new query language — it implements the actual Elasticsearch REST API: indices, documents, bulk operations, search, aggregations, mappings, kNN, scroll, reindex, and the _cat endpoints are all covered. The payoff is direct: Kibana and the official client libraries connect without modification, and any tooling already built around Elasticsearch doesn't need to be rewritten.

The project also clearly documents one protocol-level behavioral difference worth knowing before you point export tooling at it: Scroll in XERJ is a bounded, up-front snapshot, not a segment-walking cursor — if a query's actual hit count exceeds the snapshot window, XERJ returns a 400 and refuses the request rather than silently truncating the result. For result sets of any size, you need search_after instead. This "fail loudly rather than silently truncate" design choice, paired with a 1,369-case conformance suite that runs on every single commit (1,366 currently passing, with the gaps listed openly in the repo), reflects a responsible stance toward the word "compatible" — not just claiming it, but continuously verifying it and publishing exactly where the gaps remain.

The project also ran a head-to-head benchmark against Elasticsearch 8.13.4 across ingest, full-text search, aggregations, vector search, and reads under a concurrent write flood. The latest closed-loop run scores 55 wins, 26 ties, and 4 losses, including a 1.72x ingest throughput advantage and a 1.61x smaller on-disk footprint. All four losses converge on the same gap — read p99 latency while a high-rate writer is running — and instead of leaving that out, the project wrote up a dedicated findings doc explaining exactly where that shortfall shows up.

Resources

• 🌟 GitHub: xerj-org/xerj
• 🌐 Website: xerj.org
• 📖 Docs: xerj.org/docs
• 🍳 Recipes: xerj.org/docs/recipes
• 🤖 Agent-facing install guide: xerj.org/llms.txt
• 📊 Benchmarks: xerj.org/benchmarks
• 🔒 Security model: docs/SECURITY_MODEL.md
• 🗺️ Roadmap: ROADMAP.md

Summary

XERJ answers a question that's gotten sharper as AI agents scale up: once a codebase is too large for an agent's context window to hold, "reading files" isn't just slow — it's structurally not viable, and the real fix is letting the agent query for exactly the passage it needs instead of trying to optimize how fast it reads.

Three things worth noting:

"Reference Coding" turns the abstract idea of retrieval-augmented generation into a measurable, concrete coding-agent workflow, not just a talking point. Plenty of projects discuss "retrieval augmentation" at the level of principle; XERJ backs it with a controlled experiment (2.7x fewer tokens at the same solve rate) and honestly flags where the win shrinks — on popular public libraries the model already knows. That willingness to spell out when the approach actually helps is worth more trust than a bare performance number.

Choosing Elasticsearch compatibility over a custom interface is a pragmatic, almost conservative engineering call. It means the project's moat isn't "inventing a new protocol" — it's "implementing an already widely-integrated protocol well, with far fewer resources." A single Rust binary, no JVM dependency, 1.72x ingest throughput — all of that stacks directly on top of tooling users already know, so migration cost approaches zero.

Publicly disclosing its own shortcomings — four benchmark losses, three failing conformance tests, explicit boundaries on when the headline feature doesn't help — runs through the entire documentation style. That "write down the negative results too" engineering culture is relatively rare among performance-forward projects, and it's exactly the signal worth checking when judging whether a project's benchmark numbers can actually be trusted.

If your coding agent keeps hitting context-window limits in unfamiliar codebases, or you're building an internal RAG system for your team without wanting to bolt on yet another vector database, XERJ's approach — one binary, compatible with a protocol you already know, auto-indexed — is worth a serious try.

Explore PrimeSkills — a curated marketplace of AI agents and skills, each validated against real enterprise workflows. No hype, just what actually works.

Visit my personal site for more insights and interesting products.