Codebase Knowledge Base (13): Evaluation — How to Know Your Knowledge Base Is Good Enough

Why Recall@5 Isn't Enough

Article 03 set a precedent: test the vector path on 30 retrieval questions, Recall@5 = 0.958. Every subsequent article with retrieval experiments used similar reporting.

Recall@5 is a reasonable starting metric — but it has three blind spots.

Blind spot 1: Precision. Recall@5 asks 'did the correct answer appear in the top 5 results?' But code retrieval often targets 'precisely locate that one function.' Appearing 5th versus 1st scores identically in Recall@5, but the practical difference is large — first place means direct hit, fifth place means manual scanning.

Blind spot 2: Task type. Article 08 established a three-path architecture, with different paths serving different tasks: vector for semantic exploration, graph for structural traversal, symbol for exact matching. Which path does a Recall@5 dataset evaluate? Combined evaluation of all three, or separate? Combined evaluation masks each path's specific weaknesses.

Blind spot 3: Impact analysis. One of the core values of a codebase knowledge base is 'if I change this function, what else does it affect?' The correct answer to this question is a set (all upstream callers), and evaluation must check not just 'did it return relevant results' but 'did it miss any important callers?' That's a recall problem — but not Recall@5.

Four-Dimensional Evaluation Metrics

Given codebase knowledge base characteristics, four purpose-built metrics:

Metric 1: Symbol Location Accuracy

Definition: For a set of 'where is this feature implemented' queries, the percentage where the correct answer (target function) appears in first place.

Why Top-1 instead of Top-5: Code retrieval 'success' means finding that specific function. Appearing in third place means manual filtering is still required. Top-1 accuracy measures direct-hit capability, which has the largest impact on actual engineering usability.

Evaluation dataset construction:

Sample queries:
Q1: "how to insert a new document into LightRAG"
A1: lightrag/lightrag.py::ainsert (function, line 1428)
 
Q2: "what query modes does LightRAG support"
A2: lightrag/base.py::QueryParam (class, line 83)
 
Q3: "where is document chunking strategy decided"
A3: lightrag/parser/routing.py::resolve_chunk_options
 
Q4: "what cleanup operations does deleting a document trigger"
A4: lightrag/lightrag.py::adelete_by_doc_id

Based on this series' experimental data: article 03's vector Recall@5 = 0.958, but Top-1 accuracy typically lands at 0.75-0.85 — the answer is in the top 5, but first-place rate is lower.

Metric 2: Semantic Search Recall

Definition: For a set of queries, the percentage where the correct answer appears in the top K results. K is typically 5 or 10.

Difference from Recall@5: The evaluation target isn't full-set retrieval — it specifically focuses on vocabulary mismatch scenarios: the user says 'file parsing,' the code calls it 'document ingestion pipeline'; the user says 'caching mechanism,' the code uses 'KV storage with TTL.'

These queries most clearly demonstrate the value of the vector path: BM25 fails at vocabulary mismatch, while vector embeddings bridge the lexical gap.

Key dataset construction principle: queries must deliberately use different terminology than the code. Otherwise BM25 can answer correctly, which doesn't test the vector path's contribution.

Strong test cases (vocabulary mismatch):
Q: "document deduplication logic" → A: compute_mdhash_id (operate.py)
Q: "LLM request rate control" → A: priority_limit_async_func_call (utils.py)
Q: "knowledge graph node merging" → A: _merge_nodes_then_upsert (operate.py)
 
Weak test cases (vocabulary directly matches — BM25 can do this too):
Q: "priority limit async func" → A: priority_limit_async_func_call

Metric 3: Impact Analysis Completeness

Definition: For a target function, given a query for 'all direct callers,' the intersection ratio between returned callers and actual callers.

This is the metric closest to real engineering value: before modifying a function, can the knowledge base tell you every affected location?

Formula:

Completeness = |returned callers ∩ actual callers| / |actual callers|

Real measurement from LightRAG (article 09): QueryParam has 19 actual callers, search_code("QueryParam") returned all 19. Completeness = 1.0 for this case.

Not all functions are this clean. BaseVectorStorage.upsert has fan_in = 268 — if the tool has a return limit (say limit=50), it would miss 218 callers and completeness would drop sharply. That's not a recall algorithm problem — it's a tool configuration problem. But evaluation surfaces that configuration gap.

Metric 4: History Query Hit Rate

Definition: For a set of 'why is this code written this way' queries, whether FILE_CHANGES_WITH edges or detect_changes historical data provide meaningful signals.

This is the hardest metric to quantify, since 'meaningful signal' is inherently subjective. In practice, binary scoring works:

1 = query results include directly relevant historical change information
0 = query results are empty or irrelevant

For priority_limit_async_func_call: query the historical change pattern of utils.py, get 'high-frequency change file' — hit, score 1.

For base.py::QueryParam: FILE_CHANGES_WITH shows base.py ↔ lightrag.py has strong coupling — hit, score 1.

Three Sources for Evaluation Datasets

When building evaluation datasets, codebase knowledge bases have a natural advantage over document RAG: the code itself is the annotation source.

Source 1: Extracted from test files

Test files are the best documentation of 'what this function should do.' For each test function, you can reverse-engineer the corresponding production retrieval question:

# tests/test_query.py
def test_hybrid_query():
    param = QueryParam(mode="hybrid")
    result = rag.query("what is LightRAG", param)
    ...

From this test, automatically generate:

• Query: 'LightRAG hybrid query mode'
• Expected answer: lightrag/base.py::QueryParam

LightRAG has over 200 test files — each is a potential evaluation data source.

Source 2: Extracted from git commit messages

Git commit messages frequently contain 'which function had what problem fixed' — reversible into queries:

commit: "fix: priority_limit_async_func_call deadlock when worker timeout"
→ Query: 'where is the concurrent LLM call timeout deadlock handled'
→ Expected answer: lightrag/utils.py::priority_limit_async_func_call

The characteristic of this dataset type: it tests 'real questions real engineers would ask,' which is more representative of actual usage than manually constructed questions.

Source 3: Manually constructed hard cases

The first two sources tend to cover 'parts of the codebase with explicit documentation.' But the scenarios where engineers most need the knowledge base are precisely those without documentation, where understanding requires reading the code. These need manual construction:

Hard case examples:
Q: "do SDK users and REST API users go through different code paths for document processing?"
A: Yes — SDK goes through ainsert (F-only), REST API goes through apipeline_enqueue_documents
   Finding the answer means locating the branch point between the two paths
 
Q: "changing a Kafka message format — which consumers are affected?" (cross-repo)
A: Requires cross-repo analysis, finding CROSS_ASYNC_CALLS edges

These cases don't need to be the majority of the evaluation set (around 20% is sufficient), but they're the dividing line between 'good enough' and 'genuinely useful.'

Reference Baselines for Four-Dimensional Metrics

Based on experimental data from the preceding twelve articles:

How to read this table:

Semantic search recall is already high at the weak baseline (0.90+), because article 03 proved that AST-chunked vector retrieval is already quite strong. The three-path improvement shows up mainly in Top-1 accuracy and impact analysis — the two scenarios where vector retrieval alone performs poorly.

History Query Hit Rate is a four-path-only metric measuring whether the knowledge base can answer 'why' questions. The first three paths score 0 on it.

Wiring Evaluation into CI/CD

Evaluation isn't a one-time exercise — it's continuous monitoring. Code evolves and the knowledge base updates with it. After each incremental update, running a quick evaluation confirms that the update didn't degrade retrieval quality.

Minimal CI integration:

# Pseudocode — not runnable as-is
# Triggered in CI pipeline (after each PR merge)
 
def run_kb_quality_check():
    # Sample 50 questions from the annotation set
    # (random sample, covering all four metric types)
    questions = sample(EVAL_DATASET, n=50)
    
    results = {
        "symbol_accuracy": [],
        "semantic_recall": [],
        "impact_completeness": [],
        "history_hit": []
    }
    
    for q in questions:
        result = query_knowledge_base(q.query, path=q.path)
        results[q.metric_type].append(evaluate(result, q.expected))
    
    scores = {k: mean(v) for k, v in results.items()}
    
    # Alert thresholds
    THRESHOLDS = {
        "symbol_accuracy": 0.80,
        "semantic_recall": 0.90,
        "impact_completeness": 0.85,
        "history_hit": 0.65,
    }
    
    for metric, score in scores.items():
        if score < THRESHOLDS[metric]:
            alert(f"Knowledge base quality degraded: {metric} = {score:.2f}")
    
    return scores

This approach only requires maintaining an annotation dataset (50-200 questions is enough). CI runtime is roughly 5-15 minutes — acceptable overhead.

When to rebuild the annotation dataset: Large-scale refactors can invalidate old annotation sets if target functions were renamed or moved. Good practice is to record each function's git hash in the annotation set. If a target function no longer exists in the new version, automatically mark that question as 'stale' — it needs human update.

Series Retrospective: What Thirteen Articles Covered

A complete review at the final article.

This series began with one concrete question: how can engineers build a retrievable knowledge system for a large codebase?

Part 1 (Theory and Tools): Articles 01-02

Established two foundational understandings:

• The core value of a codebase knowledge base isn't 'code generation' — it's 'code understanding': answering 'where is this,' 'what does it affect,' 'why is it this way'
• Capability boundaries of existing tools: plain-text embedding, AST analysis, and knowledge graphs each have strengths, and each alone has blind spots

Part 2 (Core Technology): Articles 03-08

Conclusions driven by experimental data:

• Article 03: AST function-level chunking Recall@5 = 0.958 — the ceiling for the vector-only path
• Article 04: Four chunking strategies (F/R/V/P) — chunking granularity directly determines retrieval quality
• Article 05: Graph and vector paths are not substitutes — Q8 (cross-file structural questions) proved the graph path is irreplaceable
• Articles 06-07: Structural embeddings and hybrid search are marginal improvements; the core problem is 'routing to the wrong path,' not 'the path isn't strong enough'
• Article 08: Three-path architecture — vector/graph/symbol as orthogonal paths, routed by query intent

Part 3 (Engineering Practice): Articles 09-13

From lab to production:

• Article 09: codebase-memory-mcp three-path practice on a real project (LightRAG, 20,674 nodes)
• Article 10: Incremental update three-tier decision tree — 80% of commits can be skipped at tier one
• Article 11: Cross-repo analysis — 0 edges is a meaningful answer; LightRAG × graphrag = parallel alternatives
• Article 12: Git history is the fourth path — 465 FILE_CHANGES_WITH edges reveal implicit coupling
• Article 13: Evaluation framework — four-dimensional metrics, dataset construction, CI monitoring

The thread that runs through all thirteen:

This series has been making the same point throughout — code understanding is multi-dimensional, and no single signal covers all question types. The vector path is strong when vocabulary matches, the graph path is irreplaceable for structural traversal, the symbol path is unambiguous for precise location, and the history path is unique for understanding evolutionary context.

A good codebase knowledge base doesn't pick the strongest single path — it maps out each path's capability boundary clearly, then routes by question type.

Summary

The Codebase Knowledge Base series ends here.

Thirteen articles covered the full engineering lifecycle: from theory to tooling, from experiments to production, from single-repo to cross-repo, from building to maintaining.

If there's one sentence that captures the core conclusion of the whole series:

The quality of a codebase knowledge base depends not on how powerful the embedding model or how large the graph is — it depends on whether each path's capability boundaries are clearly understood, and whether the routing logic is correct.

Four retrieval paths, a three-tier decision tree, four-dimensional evaluation — none of this is complex engineering. It's what naturally emerges when you think clearly about the structure of the 'code understanding' problem.

Follow me at dongqi.dev for more LLM engineering content.

At PrimeSkills, we help engineering teams take codebase knowledge bases from prototype to production — including four-path retrieval architecture design, evaluation framework setup, and CI/CD integration. If your team is considering building a knowledge system for a large codebase, reach out.

primeskills.dev