Enterprise Knowledge Base (00): Build the Evaluation Dataset Before Writing a Single Line of Code

Why Build the Test Set First

This series has one goal: compare six open-source RAG frameworks (LightRAG, GraphRAG, HippoRAG, HyperGraphRAG, RAG-Anything, gbrain) and produce selection guidance.

But "comparison" has a prerequisite: the same questions, the same scoring criteria. If every article uses different documents and different questions, the results aren't comparable — you tested LightRAG on API docs, GraphRAG on papers, and called it a benchmark. That's not comparison, that's each system playing to its strengths.

So before running the first framework, build the test set. This article covers that process end to end.

Why Not Just Use BEIR

The first instinct is to find an existing standard dataset. The most commonly cited benchmark in RAG evaluation is BEIR — 18 subsets covering question answering, fact verification, medical literature, and more.

But BEIR has a fundamental problem: it's built for academic retrieval tasks, not enterprise knowledge base use cases.

Testing enterprise RAG with BEIR is like using college entrance exam questions to evaluate job candidates — the question distribution doesn't match, so the scores carry no signal.

The right approach: synthesize domain-specific questions from documents that match the target distribution.

Choosing the Source Documents

The test set needs to represent "enterprise technical documentation" as a category. Two sets of official open-source project documentation were selected:

LightRAG official docs (13 Markdown files):

• API server configuration, Docker deployment, multi-site deployment
• File processing pipeline, chunking strategies, custom parser development
• Milvus configuration, offline deployment, role-specific LLM configuration

graphrag official docs (17 Markdown files):

• Architecture design, default dataflow, input/output formats
• Index configuration, query modes, prompt tuning
• CLI usage, visualization tools

These documents have exactly the characteristics typical of enterprise technical documentation: operational steps, configuration examples, and cross-document dependencies. And since LightRAG and graphrag are themselves frameworks being evaluated in this series, using their docs as the test corpus is both self-consistent and practically representative.

Total: 30 valid documents (after filtering files shorter than 200 characters).

Designing Three Question Types

The test set needs to cover three distinct failure modes:

Type 1: Single-Hop Factual Queries (50%, 50 questions)

The answer is directly in one document — no cross-document reasoning needed.

Tests: Basic retrieval recall — given a question, can the system find the document chunk containing the answer?

Real examples:

Q: What are the two main categories of configuration settings
   in the LightRAG Docker Deployment?
A: Server Configuration and LLM Configuration
src: [DockerDeployment.md]
 
Q: What is the condition under which query/document asymmetric
   embedding is enabled in LightRAG?
A: query/document asymmetric embedding is enabled only when
   EMBEDDING_ASYMMETRIC=true is explicitly set
src: [AsymmetricEmbedding.md]

Type 2: Multi-Hop Reasoning (30%, 20 questions)

The answer requires combining information from multiple documents. Typical scenarios: cross-document comparisons, system integration understanding, end-to-end workflow tracing.

Tests: The incremental value of graph RAG and hybrid retrieval over pure vector retrieval — you found the individual documents, but couldn't assemble the complete answer.

Real examples:

Q: How does the configuration of Milvus index parameters through
   vector_db_storage_cls_kwargs facilitate a multi-site deployment
   of LightRAG?
A: The configuration allows dynamic per-instance tuning in a
   multi-site setup...
src: [MilvusConfigurationGuide.md, MultiSiteDeployment.md]
 
Q: Compare the authentication flow in LightRAG API Server with
   the role-specific LLM configuration approach...
src: [LightRAG-API-Server.md, RoleSpecificLLMConfiguration.md]

Type 3: Boundary Refusal (20%, 19 questions)

Questions that are plausible and on-topic, but whose answers are not in the documents. This is where enterprise knowledge bases most commonly fail — many systems will hallucinate a plausible-sounding but entirely wrong answer.

Tests: Refusal capability. The correct answer is "the documents don't contain this information," not a fabricated one.

Real examples:

Q: What is the cost of a premium support plan for RAG system deployments?
A: The provided documents do not contain information about this topic.
 
Q: How does the RAG system handle data privacy for users
   in the EU under GDPR regulations?
A: The provided documents do not contain information about this topic.
 
Q: What are the security protocols implemented in the DockerDeployment?
A: The provided documents do not contain information about this topic.

Synthesizing Questions with an LLM

Writing 89 questions by hand isn't practical. An LLM can do this — give it the source documents, ask it to generate questions by type, and constrain the output format in the prompt.

Single-hop prompt structure:

You are a technical documentation expert creating evaluation questions
for a RAG system.
 
Given the following technical document, generate {n} single-hop factual
questions. Requirements:
- Questions must be answerable from this document alone
- Cover key concepts, configurations, or procedures
- Output strictly as a JSON array
 
Document title: {title}
Document content: {content}
 
Output format:
[{"question": "...", "ground_truth": "...", "question_type": "single_hop"}]

Multi-hop prompt structure:

Given MULTIPLE technical documents, generate questions that require
information from at least 2 different documents to answer fully.
 
Document 1: {title_1}\n{content_1}
Document 2: {title_2}\n{content_2}
...

Boundary prompt structure:

Given the topics covered in the following documents, generate questions
about this domain that the documents do NOT answer.
This tests the system's ability to say "I don't know."
Covered topics: {covered_topics}

Problems Encountered During Generation

Problem 1: ragas TestsetGenerator timing out during transforms

The initial approach used ragas's official TestsetGenerator, which runs HeadlinesExtractor, SummaryExtractor, and other transforms on all documents before generating questions. Under local network conditions, this process made frequent LLM API calls that timed out — 30 documents, 22 minutes of running, then hung.

Final decision: abandon ragas TestsetGenerator for generation. Ragas is kept for the evaluation phase (computing Faithfulness, Answer Relevancy, etc.), not the generation phase.

Problem 2: JSON parsing failures on certain document groups

GLM-4-flash occasionally returned incomplete JSON for some documents (extra explanatory text, or missing brackets), causing json.loads to fail and those questions to be filtered out.

Affected documents: MultiSiteDeployment.md, ParserDebugCLI.md, Reproduce.md, graphrag_get_started.md, graphrag_manual_prompt_tuning.md — these documents consist mainly of configuration examples and CLI commands, which leads the LLM to output code blocks instead of JSON.

Final yield: 89 questions (target was 100; ~10% shortfall is acceptable).

Final Dataset Structure

kb-00-testset/
├── generate_testset.py          # Generation script
├── .env.example                 # Environment variable template
├── data/
│   ├── raw_docs/                # 30 source documents
│   │   ├── AsymmetricEmbedding.md
│   │   ├── DockerDeployment.md
│   │   ├── ... (LightRAG docs)
│   │   ├── graphrag_architecture.md
│   │   └── ... (graphrag docs)
│   └── output/
│       ├── testset.jsonl        # 89 evaluation questions
│       └── testset_stats.json   # Summary statistics

Each record format:

{
  "question": "What are the two main categories of configuration...",
  "ground_truth": "Server Configuration and LLM Configuration",
  "source_docs": ["DockerDeployment.md"],
  "question_type": "single_hop"
}

Dataset statistics:

{
  "total": 89,
  "single_hop": 50,
  "multi_hop": 20,
  "boundary": 19,
  "source_docs_count": 30,
  "llm_model": "glm-4-flash"
}

What This Test Set Measures

Every subsequent framework evaluation article (LightRAG, GraphRAG, HippoRAG, etc.) will use these 89 questions and report a consistent set of metrics:

The first four come from the RAGAS framework. The fifth is a custom metric for this series — it tests whether each framework honestly says "I don't know" when it should. The sixth addresses production viability.

Together, these six dimensions answer the complete selection question: is this framework good enough for my use case?

Running the Script

cd llm-in-action
 
# Copy environment variables
cp kb-00-testset/.env.example kb-00-testset/.env
# Fill in LLM_API_KEY and other settings
 
# Generate the test set (runs from any directory)
python kb-00-testset/generate_testset.py
 
# Custom question count
python kb-00-testset/generate_testset.py --size 50

Dependencies:

conda activate dev_base
pip install openai python-dotenv

Next article: the first framework evaluation — LightRAG vs QAnything: Classic Vector RAG Head-to-Head. Same 89 questions, scoring both frameworks on single-hop, multi-hop, and boundary refusal.

Follow me at dongqi.dev for more LLM engineering content.

At PrimeSkills, we help engineering teams design evaluation frameworks for enterprise knowledge bases, including domain-specific test set construction and continuous evaluation pipelines. If your team is selecting a RAG framework, reach out.

primeskills.dev