The Cost of No Metrics
How do you know when a Skill gets worse?
- Wait for user complaints — how many bad experiences happened before the first one arrived?
- Wait for someone to say "the AI feels worse lately" — no way to isolate which Skill, which dimension
- Wait for business metrics to drop — expensive to trace back
A metrics system catches degradation before users do.
The L1/L2/L3 Framework
L3 — System Health
latency, availability, token cost, error rate
collection: automatic on every call
L2 — Output Quality
format compliance, LLM-as-Judge quality score
collection: periodic sampling (daily for high-volume, weekly for low)
L1 — Business Outcome
task completion rate, adoption rate, user rating
collection: user feedback + behavior trackingLayer dependency:
L3 healthy → L2 quality → L1 value
L3 timeouts → L2 output truncated → L1 task fails
L3 ok, L2 poor → L1 users don't adopt
All three healthy → Skill delivers real valueWhen an alert fires, start from L3 and work up. Diagnosing from L1 down is much slower.
Demo Design
Test subject: rnd-technical-writer — given a topic, write a Markdown technical article.
6 calls with mixed Chinese and English:
| ID | Input (truncated) | Language |
|---|---|---|
| T01 | Python asyncio event loop internals | EN |
| T02 | Redis cache penetration, breakdown, avalanche | CN |
| T03 | Docker multi-stage builds | EN |
| T04 | LangGraph state management tutorial | CN |
| T05 | HTTP/2 multiplexing | EN |
| T06 | Rust ownership model for Python developers | CN |
L2 format check (rule-based, no LLM):
def check_format(article: str) -> tuple[bool, list[str]]:
issues = []
if "---" not in article[:300]:
issues.append("missing frontmatter")
if len(re.findall(r"^## ", article, re.MULTILINE)) < 3:
issues.append("fewer than 3 H2 sections")
if "```" not in article:
issues.append("no code block")
if len(article.split()) < 200:
issues.append(f"too short: {len(article.split())} words")
return len(issues) == 0, issuesL2 quality scoring (LLM-as-Judge, weighted):
technical_accuracy × 0.35
depth × 0.25
clarity × 0.20
practical_value × 0.20Run Results
Step 1: L3 Collection
[T01] Python asyncio... ✓ 52.9s ~1515 tokens
[T02] Redis caching... ✓ 40.2s ~470 tokens
[T03] Docker multi-stage... ✓ 33.5s ~1312 tokens
[T04] LangGraph tutorial... ✓ 50.6s ~996 tokens
[T05] HTTP/2 multiplexing... ✓ 39.9s ~1264 tokens
[T06] Rust ownership... ✓ 39.4s ~541 tokensStep 2: L2 Scoring
[T01] ✓ format ok acc=4 dep=4 cla=5 pra=4 → 4.20/5
[T02] ✗ missing frontmatter; no code block; too short: 71 words
acc=4 dep=3 cla=4 pra=3 → 3.55/5
[T03] ✓ format ok acc=4 dep=4 cla=5 pra=4 → 4.20/5
[T04] ✓ format ok acc=4 dep=3 cla=4 pra=4 → 3.75/5
[T05] ✓ format ok acc=4 dep=3 cla=5 pra=3 → 3.75/5
[T06] ✗ missing frontmatter; too short: 149 words
acc=4 dep=3 cla=4 pra=3 → 3.55/5Health Dashboard
══════════════════════════════════════════════════════════════════════
Skill Health Dashboard: rnd-technical-writer
══════════════════════════════════════════════════════════════════════
L3: System Health
Availability 100.0% >99% ✓
P90 Latency 50.6s <30s ✗
P99 Latency 50.6s <60s ✓
Avg Tokens/call 1016 (budget)
L2: Output Quality
Format Compliance 66.7% >95% ✗
Quality Score (L-J) 3.83 >3.8/5 ✓
L1: Business Value
Task Completion 83.3% >75% ✓
Adoption Rate 66.7% >60% ✓
Avg User Rating 3.50 >4.0/5 ✗
Alerts:
🟡 [WARNING] P90 latency > 30s current=50.6s threshold=30s
🟡 [WARNING] Format compliance < 95% current=66.7% threshold=95%
🟡 [WARNING] Avg rating < 4.0 current=3.50 threshold=4.0Three Findings
Finding 1: P90 at 50.6s, Every Call Over Threshold
All 6 calls exceeded the 30s threshold; the fastest was T03 at 33.5s. Generating a full technical article takes glm-4-flash 30–50 seconds — visible waiting for the user.
Without metrics, the team hears "the AI feels slow." With P90 data, you can say: "P90 is 50.6s, 67% above the 30s target. Decision: switch models or add streaming output."
Next steps:
- Evaluate models with lower generation latency
- Add streaming output so users see text appearing rather than waiting for the full response
- Or adjust the threshold to 60s if this generation time is acceptable for the use case
Finding 2: Chinese Requests Fail Format Checks — English Passes 100%
T01, T03, T05 (English) all passed format checks. T02 and T06 (Chinese) both failed:
- T02: missing frontmatter, no code block, "71 words" (the
split()word count is nearly meaningless for Chinese text) - T06: missing frontmatter, "149 words" (same issue)
Two problems stacked:
split()word count fails on Chinese — no spaces between words, sosplit()returns 71 tokens for a 700-character article- The model skips frontmatter on some Chinese requests — English prompts almost always produce it
The Skill prompt is written in English. Format requirements weren't restated for Chinese output, so the model treated them as optional. Fix:
## Output Requirements
Regardless of the request language (Chinese or English), the output MUST include:
- YAML frontmatter (title, description, tags)
- At least 3 H2 sections
- At least 1 fenced code blockFinding 3: Quality Score at 3.83 — Barely Above the Threshold
Average quality score 3.83/5, threshold 3.8. It passed, 0.03 above the line. Without a threshold, that number reads as "fine."
T02 and T06 both scored 3.55, pulling the average down. Their depth and practical_value dimensions both hit 3 instead of 4 or 5 — the Chinese articles were shorter and less detailed, so the judge correctly scored them lower.
The two L2 signals confirmed each other: format check found "missing frontmatter, no code block," quality score found "depth=3, practical_value=3." Both pointed to the same gap in the Skill prompt's coverage of Chinese output.
Alert Thresholds Reference
# L3
availability_7d < 99% → CRITICAL: investigate immediately
p90_latency > 30s → WARNING: check model/endpoint status
p90_latency > 60s → CRITICAL: service degraded
# L2
format_compliance < 95% → WARNING: sample failing outputs
quality_score_7d < 3.8 → WARNING: review low-scoring calls
quality_score_7d < 3.5 → CRITICAL: consider rolling back Skill version
# L1
task_completion_rate_delta < -10% (weekly) → CRITICAL: user interviews + regression test
adoption_rate < 60% → WARNING: investigate adoption blockers
avg_rating < 4.0 → WARNING: cross-reference with L2 dataImplementation Roadmap
Step 1 (no tools needed, do now):
□ Inventory your current Skills
□ Manually estimate call frequency and subjective quality for each
→ This is your L1 baseline
Step 2 (1–2 weeks):
□ Log latency and token count on each Skill call
□ Get real L3 baseline numbers
Step 3 (2–4 weeks):
□ Build an L2 evaluation set for your top 3 Skills
□ Run weekly sampling, track quality score trend
Step 4 (monthly):
□ Add a 👍/👎 feedback prompt after each Skill output
□ Roll up all three layers into a monthly Skill health reportSummary
- P90 alerts turn "feels slow" into a number: 50.6s vs 30s threshold, directly actionable — switch models, add streaming, or adjust the target
- Format compliance exposed a language blind spot: English 100%, Chinese failing on frontmatter; the Skill prompt covered English behavior implicitly but never stated it explicitly for Chinese
- Quality score 3.83 barely passed, L2 dual signals confirmed each other: format check found structure issues, quality score found depth issues — both pointed to the same gap in the Skill prompt's coverage of Chinese output
References
- Langfuse documentation — Skill call tracing
- Full demo code: skill-04-metrics
Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.
Find more useful knowledge and interesting products on my Homepage