Enterprise Knowledge Base (02): Classic Vector RAG Benchmark — QAnything vs LightRAG

Starting Point

The previous article built the unified test set: 89 questions — 50 single-hop factual queries, 20 multi-hop reasoning, 19 boundary refusal — sourced from LightRAG and graphrag official documentation.

This article's job: run both frameworks against the same test set and report the numbers.

But before the numbers, the deployment process — because deployment complexity is itself a selection criterion.

Deployment Comparison

LightRAG: pip install and done

pip install lightrag-hku

No Docker, no database services. Knowledge graph and vector index live in local files:

rag_storage/
  ├── graph_chunk_entity_relation.graphml   # knowledge graph
  ├── vdb_chunks.json                        # document vectors
  ├── vdb_entities.json                      # entity vectors
  └── vdb_relationships.json                 # relationship vectors

Initialization:

from lightrag import LightRAG, QueryParam
from lightrag.utils import EmbeddingFunc
 
rag = LightRAG(
    working_dir="./rag_storage",
    llm_model_func=llm_func,
    embedding_func=EmbeddingFunc(
        embedding_dim=1024,
        max_token_size=8192,
        func=embed_func,
    ),
)
await rag.initialize_storages()   # required in v1.5.x
await rag.ainsert(document_text)
result = await rag.aquery(question, param=QueryParam(mode="mix"))

One v1.5.x gotcha: initialize_storages() must be called before anything else, or you get PipelineNotInitializedError.

QAnything: 5 Docker services

QAnything v2 requires a full infrastructure stack:

services:
  elasticsearch       # keyword retrieval (BM25)
  etcd                # Milvus metadata store
  minio               # Milvus object storage
  milvus-standalone   # vector database
  mysql               # document and KB metadata
  qanything_local     # main service (embedding + rerank + API)

Startup:

cd QAnything
mkdir -p volumes/es/data && chmod 777 -R volumes/es/data
docker compose -f docker-compose-linux.yaml up -d
# Wait for "qanything后端服务已就绪!" in logs

Pitfalls Encountered

Three real problems during deployment, each worth documenting.

Pitfall 1: QAnything container doesn't use GPU by default

The machine has an RTX 3060, but the container startup log always shows:

embedding和rerank服务将在CPU上运行
(embedding and rerank running on CPU)

Root cause: the qanything_local service in docker-compose-linux.yaml has no GPU resource configuration. Worse, that log line is a hardcoded string in entrypoint.sh — it prints regardless of whether GPU is actually available.

Fix 1: Add GPU resource to the compose file:

qanything_local:
  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
            device_ids: ['0']
            capabilities: [gpu]

Fix 2: Install NVIDIA Container Toolkit (Docker can't see the host GPU without this bridge):

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Fix 3: entrypoint.sh doesn't pass --use_gpu when starting the embedding/rerank servers:

# Before
nohup python3 -u qanything_kernel/dependent_server/embedding_server/embedding_server.py > ...
 
# After
nohup python3 -u qanything_kernel/dependent_server/embedding_server/embedding_server.py --use_gpu > ...
nohup python3 -u qanything_kernel/dependent_server/rerank_server/rerank_server.py --use_gpu > ...

After all three fixes: GPU VRAM jumped from 1.4GB to 8.6GB, document processing speed went from < 1 doc/second to roughly 1-2 docs per 15 seconds.

Pitfall 2: Milvus crashes under memory pressure

In CPU mode, the QAnything container consumed 27GB of RAM, causing Milvus standalone's etcd lease to expire and the process to exit:

"etcdserver: requested lease not found"
"connection lost detected, shuting down"

All documents got stuck in gray (queued for vectorization) and never progressed.

Fix: Clear Milvus and etcd persistent data directories before restart. The stale etcd session entries cause it to crash again immediately if not cleared:

docker compose -f docker-compose-linux.yaml down
rm -rf volumes/milvus volumes/etcd volumes/mysql
mkdir -p volumes/milvus volumes/etcd volumes/mysql
docker compose -f docker-compose-linux.yaml up -d

Pitfall 3: user_id concatenation makes Web UI show nothing

QAnything's server-side handler appends a user_info suffix to every user_id:

# handler.py
user_info = safe_get(req, 'user_info', "1234")   # default "1234"
user_id = user_id + '__' + user_info              # stored user_id

If the script sends user_id=zzp__1234, the stored ID becomes zzp__1234__1234. The Web UI's actual user is zzp__1234 — they're different users, so the Web UI can't see the API-created knowledge base.

Fix: Send user_id=zzp from the script. After server-side concatenation it becomes zzp__1234, matching the Web UI.

Evaluation Configuration

Both frameworks used identical LLM and test set:

Results

Core metrics

Note: answer match uses Jaccard keyword overlap, not LLM judge. Both frameworks produce longer answers than the ground truth (with explanations), so Jaccard values are low in absolute terms — only useful for comparison, not as absolute quality scores.

Answer quality

Same question, real outputs side by side:

Single-hop: What is the condition under which query/document asymmetric embedding is enabled?

Ground Truth: enabled only when EMBEDDING_ASYMMETRIC=true is explicitly set
 
LightRAG:   Query/document asymmetric embedding in LightRAG is enabled only when
            the EMBEDDING_ASYMMETRIC setting is explicitly set to true...
            [correct, concise]
 
QAnything:  ## Inferred Answer Section
            According to the reference information, query/document asymmetric
            embedding in LightRAG is enabled only when...
            [correct, but wrapped in Markdown section headers]

Both answer correctly. LightRAG's output is cleaner — QAnything's system prompt produces structured headers (## Inferred Answer Section) that make answers feel verbose.

Boundary question: How does the RAG system handle data privacy for EU users under GDPR?

LightRAG:   The Retrieval-Augmented Generation (RAG) system, as implemented in
            LightRAG, handles data privacy for users in the EU under GDPR...
            [no refusal — uses LLM's own knowledge to fabricate a plausible answer]
 
QAnything:  抱歉，检索到的参考信息并未提供任何相关的信息，因此无法回答。
            (The retrieved reference information contains no relevant information,
            therefore cannot answer.)
            [correct refusal]

Boundary refusal is where QAnything clearly wins. Its system prompt has explicit rules: if retrieved context is irrelevant, respond with a fixed refusal phrase. LightRAG's mix mode prioritizes retrieving related entities from the knowledge graph and tries to reason even when documents contain no answer — which produces hallucinations on boundary questions.

Latency breakdown

LightRAG: P50=13.9s, P90=19.4s, min=8.7s, max=30.8s
QAnything: P50=39.2s, P90=52.2s, min=19.6s, max=62.9s

QAnything is slower for two reasons:

1. Rerank step: every query runs a cross-encoder re-ranking pass, which is an extra model inference call
2. More complete retrieval: QAnything retrieved real source documents on 100% of queries — longer context means longer LLM processing time

LightRAG's mix mode runs one knowledge graph query and one vector query, merges them, and hands off to the LLM. When the graph traversal is narrow, it's fast; when it's wide, it can be slow.

Which One to Pick

Lean toward LightRAG if:

• You need to validate a RAG approach quickly without infrastructure overhead
• Your team can't operate Milvus/ES/MySQL in production
• Documents have complex cross-document relationships that benefit from graph traversal
• Latency matters — LightRAG P90 is ~2.7× faster than QAnything

Lean toward QAnything if:

• Boundary refusal accuracy is important (2.5× higher refusal rate in this test)
• You have Chinese documents — BCE embedding is optimized for Chinese text
• Non-technical users need a Web UI to upload documents
• You need BM25 + vector hybrid retrieval in production

What this evaluation didn't cover:

• Large-scale documents (10,000+)
• Chinese document retrieval quality (this test set is all English)
• Knowledge base update speed and stability
• QAnything's PDF and table parsing capabilities (only Markdown in this test)

These will come up in later articles.

Evaluation Code

Full code in llm-in-action/kb-02-lightrag-eval/ and llm-in-action/kb-02-qanything-eval/.

LightRAG evaluation core:

rag = LightRAG(working_dir=STORAGE_DIR, llm_model_func=llm_func,
               embedding_func=EmbeddingFunc(embedding_dim=1024, func=embed_func))
await rag.initialize_storages()
await rag.ainsert(doc_content)
answer = await rag.aquery(question, param=QueryParam(mode="mix"))

QAnything evaluation core:

# Create KB
kb_id = api_post("new_knowledge_base", {"user_id": USER_ID, "kb_name": KB_NAME})["data"]["kb_id"]
api_post("upload_files", data={"user_id": USER_ID, "kb_id": kb_id}, files={"files": fp})
 
# Wait for vectorization (poll until status=green)
while gray_count > 0:
    time.sleep(15)
    status_count = api_post("list_files", ...)["data"]["status_count"]
 
# Query
result = api_post("local_doc_chat", {
    "user_id": USER_ID, "kb_ids": [kb_id], "question": question,
    "model": LLM_MODEL, "api_base": LLM_BASE_URL, "api_key": LLM_API_KEY,
    "streaming": False
})

Next article: GraphRAG vs HippoRAG — Multi-Hop Reasoning with Graph-Augmented RAG. Same 89 questions, with focus on the multi-hop improvement margin and the time and cost of building a knowledge graph.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage