Knowledge Share
Technical articles, tutorials, and insights
AI Evaluation Series (01): Why AI Evaluation Is Hard — Uncertainty, Subjectivity, and Multiple Dimensions
AI evaluation differs from traditional software testing in three fundamental ways: non-deterministic outputs (same input, different results on different runs), subjective quality (no single correct answer), and multi-dimensional quality (accurate, relevant, and helpful are three separate things). Four evaluation methods each carry a different cost. This article builds the conceptual framework for the series.
AI Evaluation Series (03): LLM-as-Judge — How to Use LLMs to Evaluate LLMs Correctly
Three controlled experiments to quantify LLM-as-Judge biases with real numbers. Position bias: first-position win rate 67% vs 50% baseline; same answer pair flips judgment 33% of the time when order changes. Verbosity bias: verbose version of equal-quality content scores +0.5 points higher. Framing bias: strict expert prompt and standard prompt produce identical scores on glm-4-flash. Three mitigation strategies with concrete implementations.
Skill Series (04): Skill Metrics — L1/L2/L3 Monitoring That Catches Quality Drops Before Users Do
Run 6 real Skill invocations, collect L3 data (latency, tokens), score outputs with L2 format check and LLM-as-Judge, add simulated L1 user feedback, render a health dashboard. Three alerts fire: P90 latency 50.6s (threshold 30s), format compliance 66.7% (threshold 95%), avg user rating 3.5/5 (threshold 4.0).
Skill Series (01): Skill Evaluation — How to Quantify AI Skill Quality
Run a 20-case trigger evaluation (F1=0.96), score two tasks with LLM-as-Judge, then compare two prompts in A/B. Three engineering findings from the actual data: one FP reveals a boundary ambiguity in the Skill description, a silent bug in Chinese word-count checks, and LLM judges have real limits on fine-grained comparisons.