Knowledge Share

AI Evaluation Series (04): RAG Evaluation — RAGAS Four Metrics in Practice and a Counterintuitive Finding

Using RAGAS to compare raw documents vs knowledge-distilled documents in a RAG system. Results: both versions score nearly identically (average 0.933 vs 0.930), with Context Precision and Context Recall at 1.0 for both. This counterintuitive result reveals an important RAGAS limitation: on small, simple knowledge bases, the four metrics can't distinguish document quality differences. Larger scale and more complex queries are needed to see separation. The article also covers how each metric is computed and common misconceptions.