Knowledge Share
Technical articles, tutorials, and insights
AI Evaluation Series (06): DeepEval in Practice — Enterprise Agent Evaluation Suite
Running DeepEval on the same customer support Agent: AnswerRelevancy (avg 0.767, 60% pass), Faithfulness (avg 0.600, 40% pass), ToolCorrectness (20% pass — Agent's failure to call tools is the root cause). Key focus: the paradigm difference between DeepEval and RAGAS. RAGAS is a batch analysis tool; DeepEval is a CI quality gate tool. They solve different problems. Also covers how to connect glm-4-flash as the DeepEval Judge LLM.
AI Evaluation Series (07): Custom Benchmarks — From Business Scenarios to Evaluation Sets
Public benchmarks fail in three situations: the business domain is too specialized, data can't leave the organization, or you need continuous quality tracking. This article covers the complete custom benchmark construction workflow: scenario definition, question generation, ground truth annotation, difficulty stratification, and version control. Using enterprise document Q&A as an example, building a set covering Easy/Medium/Hard difficulty layers, and explaining why evaluation sets themselves need version control.
AI Evaluation Series (08): Evaluation CI/CD — Continuous Quality Gates
Integrating evaluation into CI so every code or Prompt change gets a quality judgment before merge. Two-tier strategy: fast evaluation (10 core cases, < 3 minutes) on every commit, full evaluation (100+ cases, weekly trend tracking). Focus on quality gate design: which metric drops block PRs vs alert only; and how to track Delta between two versions so reviewers see change, not just absolute score.
AI Evaluation Series (05): Agent Evaluation — Tool Call Accuracy and Trajectory Quality
Systematic evaluation of a customer support Agent (3 tools, 15 test cases). Results: tool name accuracy 73%, parameter accuracy 100%, step efficiency 0.73x. All 4 failures are 'should have called a tool but didn't' — not 'called the wrong tool.' The bottleneck is tool triggering logic, not tool execution. Trajectory quality scored 3.73/5 by LLM-as-Judge, with multi-step and edge cases performing slightly worse.
AI Evaluation Series (01): Why AI Evaluation Is Hard — Uncertainty, Subjectivity, and Multiple Dimensions
AI evaluation differs from traditional software testing in three fundamental ways: non-deterministic outputs (same input, different results on different runs), subjective quality (no single correct answer), and multi-dimensional quality (accurate, relevant, and helpful are three separate things). Four evaluation methods each carry a different cost. This article builds the conceptual framework for the series.
AI Evaluation Series (02): Metric Design — From Business Goals to Measurable Indicators
How to decompose 'AI responses should be good' into specific metrics. The L1/L2/L3 framework: L1 business outcomes (task completion rate, adoption rate), L2 output quality (accuracy, relevance, completeness), L3 system health (latency, token cost). Focus on metric selection across four common scenarios, and three traps: measuring only L3, using BLEU/ROUGE for semantic quality, and setting thresholds by instinct.
AI Evaluation Series (03): LLM-as-Judge — How to Use LLMs to Evaluate LLMs Correctly
Three controlled experiments to quantify LLM-as-Judge biases with real numbers. Position bias: first-position win rate 67% vs 50% baseline; same answer pair flips judgment 33% of the time when order changes. Verbosity bias: verbose version of equal-quality content scores +0.5 points higher. Framing bias: strict expert prompt and standard prompt produce identical scores on glm-4-flash. Three mitigation strategies with concrete implementations.
AI Evaluation Series (04): RAG Evaluation — RAGAS Four Metrics in Practice and a Counterintuitive Finding
Using RAGAS to compare raw documents vs knowledge-distilled documents in a RAG system. Results: both versions score nearly identically (average 0.933 vs 0.930), with Context Precision and Context Recall at 1.0 for both. This counterintuitive result reveals an important RAGAS limitation: on small, simple knowledge bases, the four metrics can't distinguish document quality differences. Larger scale and more complex queries are needed to see separation. The article also covers how each metric is computed and common misconceptions.