AI Evaluation Series (02): Metric Design — From Business Goals to Measurable Indicators

What Happens Without Metrics

A RAG Q&A system launches. The engineers say "all tests passed" — API response time under 2 seconds, correct format, no crashes.

Two weeks later, users report "the AI often doesn't answer the actual question." Investigation reveals: retrieval recall is only 40%. More than half the user questions can't find relevant documents.

The problem was there before launch. No one had designed a "retrieval quality" metric, so no one saw it.

The L1/L2/L3 Framework

Metrics organize into three layers, each addressing a different class of question:

L1 — Business Outcome
  The end goal: is the AI system creating value?
  Examples: task completion rate, user adoption rate, user satisfaction
 
L2 — Output Quality
  The middle layer: how good is the AI's actual output?
  Examples: accuracy, relevance, completeness
 
L3 — System Health
  The foundation: is the system running stably?
  Examples: response latency, token cost, failure rate

Layer dependencies:

L3 fails → L2 degrades (timeouts cause truncated output) → L1 drops (tasks fail)
L3 healthy, L2 poor → L1 still low (users don't adopt low-quality output)
All three healthy → L1 reflects real value

Start diagnosing from L3 upward; reverse from L1 downward is much slower.

Metric Selection by Scenario

Scenario 1: Document Q&A (RAG)

L3 System health:
  Response latency P90     → < 5s
  Token cost per query     → < 3000
  Retrieval failure rate   → < 1%
 
L2 Output quality:
  Faithfulness             → no hallucinations beyond retrieved content (RAGAS)
  Answer Relevancy         → actually addresses the question (RAGAS)
  Context Precision        → fraction of retrieved content that's useful (RAGAS)
  Context Recall           → fraction of relevant content that was retrieved (RAGAS)
 
L1 Business outcome:
  Task completion rate     → fraction of users who mark "question resolved"
  Adoption rate            → fraction of users who copy/cite the AI response

The critical metric: Context Recall — the most commonly skipped and most important RAG metric. If retrieval doesn't find the relevant content, generation quality is irrelevant.

Scenario 2: Code Generation

L3:
  Response latency P90   → < 10s (code generation is slower; allow more)
  Excessive output rate  → avoid generating far more code than needed
 
L2:
  Syntax correctness     → can the output be parsed? (automatable)
  Test pass rate         → does the output pass unit tests? (automatable)
  Usability              → how much editing is needed? (human evaluation)
  Security scan pass     → does the output contain known vulnerabilities? (automatable)
 
L1:
  Adoption rate          → fraction of suggestions the user accepts
  Edit distance          → how much the user changed the suggestion (less = better)

The critical metric: Test pass rate — fully automatable, measurable per commit, a natural fit for CI integration.

Scenario 3: Document Summarization

L2:
  Faithfulness         → summary adds nothing not in the source (most important)
  Coverage             → source's key points appear in the summary
  Conciseness          → summary is noticeably shorter than source (otherwise no value)
  Readability          → LLM-as-Judge evaluation

The critical metric: Faithfulness. Summarization's greatest risk is "adding" information that wasn't in the source (hallucination) — more harmful than omitting a detail.

Scenario 4: Agent Task Completion

L2:
  Tool call accuracy   → correct tool selected, correct arguments (automatable)
  Step efficiency      → how many steps to complete the task (fewer is better)
  Trajectory quality   → is the reasoning path sound? (LLM-as-Judge)
 
L1:
  Task completion rate → did the task get done? (the core metric)
  First-run resolution → fraction completed without follow-up questions

The critical metric: Task completion rate, but define "completed" carefully — "artifact exists" versus "artifact meets quality standard" are different requirements.

Three Traps

Trap 1: Measuring Only L3

✗ Wrong:
  "API returned 200, latency 1.2s, this version can go to production."
 
Problem: L3 health doesn't mean business value. The RAG example above — L3
         completely normal, Context Recall at 40%.

Every release needs at least one L2 sampling evaluation.

Trap 2: Using BLEU/ROUGE for Semantic Quality

# Looks reasonable
rouge_score = rouge.compute(predictions=[output], references=[reference])
 
# The problem:
reference = "The capital of France is Paris."
output_1  = "Paris is the capital city of France."  # same meaning, low ROUGE
output_2  = "The capital of France is Paris, the capital."  # repetition, high ROUGE
 
# ROUGE scores output_2 higher, but output_2 is clearly worse

BLEU/ROUGE measure word overlap, not semantic correctness. For generative outputs, use LLM-as-Judge instead. BLEU/ROUGE only work for tasks with standard reference answers, like machine translation.

Trap 3: Setting Thresholds by Instinct

✗ Wrong:
  "Let's set Faithfulness > 0.8 as the quality gate."
  (Why 0.8? It sounds reasonable.)
 
✓ Right:
  Step 1: Run 100 samples, get the current baseline (e.g., mean = 0.73)
  Step 2: Manually inspect samples with Faithfulness < 0.6 — are they acceptable?
  Step 3: Set threshold based on that inspection (e.g., 0.65)
  Step 4: Document why samples below that threshold are unacceptable

Thresholds come from data distributions and business acceptability, not from alignment with round numbers.

Complete Metric Spec: Enterprise Document Q&A System

# eval_metrics.yaml
system: document-qa
version: "1.0"
 
metrics:
  l3_system_health:
    - name: response_latency_p90
      target: "< 5000ms"
      collection: trace_log
      alert_threshold: 8000ms
 
    - name: token_cost_per_query
      target: "< 2000 tokens"
      collection: llm_callback
      alert_threshold: 4000
 
    - name: retrieval_error_rate
      target: "< 1%"
      collection: error_log
 
  l2_output_quality:
    - name: faithfulness
      tool: ragas
      target: "> 0.80"
      collection: weekly_sampling (n=100)
      alert_threshold: 0.70
 
    - name: answer_relevancy
      tool: ragas
      target: "> 0.75"
      collection: weekly_sampling
 
    - name: context_recall
      tool: ragas
      target: "> 0.70"
      collection: weekly_sampling
      note: "Most important retrieval quality metric"
 
    - name: format_compliance
      tool: rule_check
      target: "100%"
      collection: every_request
 
  l1_business_outcome:
    - name: task_completion_rate
      target: "> 70%"
      collection: user_feedback_widget
      note: "Fraction of users clicking 'this solved my question'"
 
    - name: adoption_rate
      target: "> 50%"
      collection: behavior_tracking
      note: "Fraction of users who copy/cite the AI response"

Roadmap to a Working Metric System

Step 1 (no tools needed, do now):
  □ Write down the system's core business goal (what is L1?)
  □ List 3-5 most important L2 output quality dimensions
  □ Confirm L3 basics are monitored (latency, error rate)
 
Step 2 (within 1 week):
  □ Manually evaluate 50 real use cases to get baseline numbers per L2 metric
  □ Set thresholds based on that baseline — not on intuition
 
Step 3 (ongoing):
  □ Run L2 sampling eval before each release; compare to baseline
  □ Check L1 data monthly; verify it moves consistently with L2

Summary

L1/L2/L3 each have a distinct role: L3 monitors system stability, L2 evaluates output quality, L1 validates business value — missing any one layer creates a blind spot
Scenario determines critical metrics: RAG systems need Context Recall above all; code generation needs test pass rate; Agents need task completion rate — copying another system's metrics doesn't mean they fit yours
Thresholds come from data: run a baseline first, then set the threshold — don't set a threshold and then justify it with data afterward

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage