AI Evaluation Series (08): Evaluation CI/CD — Continuous Quality Gates

Prompt changes:
  Modify system prompt to reduce token cost
  → Answer Relevancy may drop 15%
  → Without CI: users complain before anyone notices
 
Knowledge base updates:
  Add new documents, update existing ones
  → Inconsistent chunking in new docs drops Context Recall
  → Without CI: weekly reports surface the problem days later
 
Model version updates:
  LLM provider silently upgrades model weights
  → Output style shifts, Faithfulness drops
  → Without CI: only discoverable after enough bad samples accumulate

# fast_eval_cases.yaml
FAST_EVAL_CASES = [
    {"id": "FE01", "scenario": "factual",    "question": "What's the refund policy?", ...},
    {"id": "FE02", "scenario": "factual",    "question": "How long is the warranty?", ...},
    {"id": "FE03", "scenario": "procedure",  "question": "How do I request an invoice?", ...},
    {"id": "FE04", "scenario": "procedure",  "question": "How do I change my address?", ...},
    {"id": "FE05", "scenario": "comparison", "question": "Gold vs Platinum membership?", ...},
    {"id": "FE06", "scenario": "multi_hop",  "question": "How much refund for ORD-004?", ...},
    {"id": "FE07", "scenario": "edge",       "question": "Do you accept Bitcoin?", ...},
    # Historically difficult cases
    {"id": "FE08", "scenario": "ambiguous",  "question": "...", ...},
    {"id": "FE09", "scenario": "multi_doc",  "question": "...", ...},
    {"id": "FE10", "scenario": "adversarial","question": "...", ...},
]

# .github/workflows/fast_eval.yml
name: Fast Quality Gate
on:
  pull_request:
    paths:
      - "prompts/**"
      - "rag/**"
      - "knowledge_base/**"
 
jobs:
  fast-eval:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - run: pip install ragas deepeval
      - name: Run fast evaluation
        run: python eval/run_fast_eval.py --output fast_eval_result.json
      - name: Check quality gates
        run: python eval/check_gates.py fast_eval_result.json
        # exits 1 on failure → blocks merge
      - name: Comment on PR
        if: always()
        run: python eval/post_pr_comment.py fast_eval_result.json

# .github/workflows/full_eval.yml
name: Full Quality Evaluation
on:
  schedule:
    - cron: "0 2 * * 1"   # every Monday at 2am
  workflow_dispatch:         # manual trigger supported
  push:
    branches: [main]
    paths:
      - "knowledge_base/**"
 
jobs:
  full-eval:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v4
      - run: pip install ragas deepeval
      - run: python eval/run_full_eval.py --output full_eval_result.json
      - run: python eval/compare_baseline.py full_eval_result.json baselines/latest.json
      - name: Update baseline if on main
        if: github.ref == 'refs/heads/main'
        run: python eval/update_baseline.py full_eval_result.json

# eval_gates.yaml
gates:
  # Block PR (critical gates)
  - metric: faithfulness
    threshold: 0.70
    mode: block
    reason: "Hallucination risk — directly impacts user trust"
 
  - metric: answer_relevancy
    threshold: 0.65
    mode: block
    reason: "Off-topic answers — core function broken"
 
  - metric: context_recall
    threshold: 0.60
    mode: block
    reason: "Missing critical information — incomplete answers"
 
  # Warn but don't block
  - metric: context_precision
    threshold: 0.70
    mode: warn
    reason: "Retrieval noise — quality impact but not critical"
 
  - metric: tool_correctness
    threshold: 0.60
    mode: warn
    reason: "Agent tool issues — needs attention but allows margin"
 
  # Delta gates (based on change, not absolute value)
  - metric: faithfulness
    max_delta: -0.05
    mode: block
    reason: "This change significantly increased hallucination rate"

def check_gates(current: dict, baseline: dict, gates: list) -> list[str]:
    failures = []
    for gate in gates:
        metric = gate["metric"]
        current_score = current[metric]
        baseline_score = baseline.get(metric)
 
        # Absolute threshold gate
        if "threshold" in gate and current_score < gate["threshold"]:
            failures.append(
                f"[{gate['mode'].upper()}] {metric}: {current_score:.3f} < {gate['threshold']}"
            )
 
        # Delta gate
        if "max_delta" in gate and baseline_score is not None:
            delta = current_score - baseline_score
            if delta < gate["max_delta"]:
                failures.append(
                    f"[{gate['mode'].upper()}] {metric} dropped {delta:+.3f} "
                    f"(threshold: {gate['max_delta']:+.3f})"
                )
    return failures

## AI Quality Evaluation Report
 
Time: 2026-07-16 14:23  |  Cases: 10  |  Duration: 2.1 min
 
| Metric | Current | Baseline | Change | Status |
|--------|---------|----------|--------|--------|
| Faithfulness | 0.87 | 0.82 | **+0.05** | ✅ Improved |
| Answer Relevancy | 0.71 | 0.79 | **-0.08** | ⚠️ Warning |
| Context Recall | 0.88 | 0.84 | **+0.04** | ✅ Improved |
| Context Precision | 0.79 | 0.81 | -0.02 | ✅ Normal range |
| Tool Correctness | 0.60 | 0.73 | **-0.13** | 🚫 Blocked |
 
**Verdict: Tool Correctness dropped 0.13, exceeding threshold (-0.05). PR blocked.**
 
Failing case:
- FE06 (order refund query): expected get_order_status + calculate_refund,
  actual: only calculate_refund called. Agent used the amount from the question
  directly instead of fetching it from the order record.
 
Suggested fix: check whether this prompt change affected multi-step tool triggering logic.

class BaselineManager:
    def should_update_baseline(self, current: dict, previous: dict) -> bool:
        # Rule 1: current results meaningfully better (avg +0.02 across metrics)
        avg_delta = sum(current[k] - previous[k] for k in current) / len(current)
        if avg_delta > 0.02:
            return True
 
        # Rule 2: baseline is stale (>30 days)
        if baseline_age_days > 30:
            return True
 
        # Rule 3: explicit major version release (manual trigger only)
        return False
 
    def update_baseline(self, current: dict) -> None:
        archive_current_baseline()   # preserve history
        save_as_new_baseline(current)

Code / Prompt / Knowledge Base Change
          ↓
Fast Eval (10 cases, < 3 min)
  Pass → PR can merge
  Fail → Block + PR comment with root cause
 
Weekly schedule
          ↓
Full Eval (100+ cases, ~1 hour)
  ↓ Compare with baseline (Delta report)
  ↓ Store in quality trend database
  ↓ Quality drop → notify team lead
  ↓ Consistent improvement → update baseline
 
Monthly
          ↓
Benchmark review
  ↓ Cases that always pass? Consider adding harder variants
  ↓ Cases that always fail? Check if it's a test problem or system problem
  ↓ New business scenarios to add?
  ↓ Update eval_dataset.yaml, bump version